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ABSTRACT 


This  thesis  research  examines  the  effectiveness  of  a  newly  developed 
cognitive  and  pedagogical  evaluation  framework  to  assess  computer-based 
instruction.  All  training  programs  must  have  comprehensive  evaluation  guidelines 
in  place  to  ensure  the  quality  of  instruction  from  the  classroom-training 
environment  to  the  virtual  training  environment  is  not  diminished.  The  application 
of  sound  cognitive  and  pedagogical  principles  helps  ensure  that  an  organization’s 
training  goals  will  be  met.  This  research  developed  a  set  of  practical  guidelines, 
or  a  template,  that  should  be  used  to  evaluate  the  cognitive  and  pedagogical 
aspects  of  any  given  computer  delivered  course  of  instruction.  This  template  is 
used  to  evaluate  the  United  States  Navy’s  newly  developed  CD-ROM  Surface 
Warfare  Officer  (SWO)  training  course.  The  SWO  course  is  the  basic 
professional  training  for  junior  Ensigns  that  is  now  contained  on  CD-ROM  and 
delivered  by  personal  computer. 
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EXECUTIVE  SUMMARY 


This  thesis  research  examines  the  effectiveness  of  a  newly  developed 
cognitive  and  pedagogical  evaluation  framework  to  assess  computer-based 
instruction.  All  training  programs  must  have  comprehensive  evaluation  guidelines 
in  place  to  ensure  the  quality  of  instruction  from  the  classroom-training 
environment  to  the  virtual  training  environment  is  not  diminished. 

Chapter  II  provides  background  information  on  human  sensory  perception, 
short  and  long  term  memory,  problem  solving,  and  learning.  Chapter  III  is  a 
review  of  sound  instructional  and  pedagogical  principles  for  developing 
Computer-based  Training  (CBT)  and  evaluative  framework  development  for 
assessing  CBT  initiatives.  Chapter  IV  is  the  analysis  of  the  strengths  and 
weaknesses  of  the  United  States  Navy’s  newly  developed  CD-ROM  Surface 
Warfare  Officer  (SWO)  training  course.  The  SWO  course  is  a  basic  professional 
course  of  instruction  for  junior  Ensigns  that  is  mostly  contained  on  CD-ROM  and 
delivered  through  the  use  of  a  personal  computer.  Chapter  V  is  the 
recommendations  that  arise  out  of  the  analysis  conducted  in  Chapter  IV. 
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I.  INTRODUCTION 


As  the  use  of  computers  to  enhance  and  replace  traditional  classroom 
instruction  increases,  evaluation  guidelines  must  be  developed  and  implemented 
to  ensure  that  the  quality  of  instruction  is  not  diminished.  Much  has  been  written 
about  why  and  how  to  successfully  implement  organizational  computer-based 
training  systems.  However,  little  has  been  mentioned  about  how  to  evaluate 
whether  specific  cognitive  and  pedagogical  principles  are  being  effectively 
employed.  The  application  of  cognitive  and  pedagogical  principles  helps  to 
ensure  that  the  organization’s  training  goals  will  be  met. 

This  research  developed  a  set  of  practical  guidelines,  or  a  template,  that 
should  be  used  to  evaluate  the  cognitive  and  pedagogical  aspects  of  any 
computer  delivered  course  of  instruction.  In  this  thesis,  the  template  is  used  to 
evaluate  the  United  States  Navy’s  newly  developed  CD-ROM  Surface  Warfare 
Officer  (SWO)  course.  The  SWO  basic  course  is  the  first  professional  course  of 
instruction  for  junior  Ensigns  that  is  mostly  contained  on  CD-ROM  and  delivered 
through  by  personal  computer. 

A.  BACKGROUND 

In  2002  the  United  States  Navy’s  Surface  Warfare  Officer  (SWO) 
community  dramatically  changed  the  way  its  newly  commissioned  junior  officers 
receive  their  introductory  specialty  training.  The  SWO  community  has  shifted  the 
delivery  method  from  traditional  classroom  and  laboratory  training  to  a  Compact 
Disk  Read  Only  Memory  (CD-ROM)  distance-learning  environment  that  is 
supplemented  by  on-the-job  training  (OJT).  The  Navy  awarded  the  contract  to 
re-engineer  the  then  current  curriculum  to  Intelligent  Decision  Systems,  Inc  (IDSI) 
in  mid-2002.  The  delivery  date  was  December  2002.  In  January  2003,  all  newly 
commissioned  Ensigns  in  Surface  Warfare  started  reporting  directly  to  a  ship, 
vice  the  Naval  Education  and  Training  Center,  Newport,  Rhode  Island,  and  were 
issued  the  new  courseware,  and  the  commencement  of  warfare  specialty 
training. 
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B.  PROBLEM  STATEMENT 


Human  perception,  memory,  and  critical  thinking  skills  are  all  cognitive 
processes  that  must  be  considered  by  anyone  that  is  involved  in  developing 
courses  of  learning,  whether  those  courses  are  taught  by  a  human  instructor  or 
learned  in  a  virtual  training  environment.  According  to  Filbert  and  Weatherspoon 
(1993),  designers  of  interactive  learning  systems  must  take  into  account  the 
appropriate  cognitive  learning  theories  and  instructional  design  paradigms  prior 
to  constructing  their  systems,  so  the  learning  systems  that  are  developed  will 
have  instructional  effectiveness  (p.1). 

Just  as  it  is  important  to  properly  design  instructional  systems,  it  is 
important  to  design  and  implement  effective  evaluation  guidelines.  Without 
effective  evaluation,  an  organization  will  not  know  if  its  training  objectives  are 
being  adequately  met.  Furthermore,  organizational  leadership  may  not  know  if 
the  training  methods  that  are  being  employed  are  the  best  techniques  available 
to  maximize  student  learning  throughout  the  organization’s  training  domain. 

Due  to  the  short  duration  between  the  letting  of  the  contract  and  the  date 
that  deliverables  were  due,  IDSI  had  to  hire  subcontractors,  and  utilize  off-the- 
shelf  products  where  available  and  applicable.  However,  it  is  possible  that  by 
doing  so,  the  program  emphasis  was  on  building  a  working  course  of  instruction 
rather  than  building  a  course  of  instruction  that  would  best  fit  the  students  needs. 

Consequently,  if  the  courseware  (instructional  material  and  delivery 
method)  does  not  adequately  meet  the  students’  needs,  then  it  does  not  meet  the 
Navy’s  needs.  If  the  Navy’s  needs  are  not  being  met  by  the  CD-ROM  training 
program,  then  other  alternatives,  including  the  possibility  of  returning  to  previous 
methods  of  instruction  need  to  be  identified  and  acted  upon. 

C.  SCOPE  OF  THESIS 

The  scope  includes:  (1)  A  relevant  review  of  human  memory,  cognitive 
models,  and  learning  theories,  (2)  a  review  of  current  methods  that  are  used  in 
the  arena  of  Information  Technology  and  Distance  Learning  to  implement  an 
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effective  computer-based  training  environment,  (3)  development  of  an  evaluative 
framework  that  is  sufficient  to  evaluate  the  cognitive  and  pedagogical  aspects  of 
Computer-Based  Training,  (4)  completion  of  a  cursory  analysis  of  the  United 
States  Navy’s  Division  Officer  classroom-based  course  of  instruction  for  Surface 
Warfare  Officers  and  an  in-depth  analysis  of  the  reengineered  CD-ROM  based 
curriculum  and  methods  of  delivery.  The  thesis  will  conclude  with  predictions  for 
the  eventual  success  or  failure  of  the  redesigned  curriculum  where  the  use  of 
Information  Technology  has  been  implemented  and  will  offer  suggestions,  if  any, 
for  the  improvement  of  the  pedagogy  and  curriculum.  At  this  time  there  are  no 
objective  data  available  for  comparing  computer-based  to  conventional 
instruction.  Accordingly,  the  course  assessment  is  based  primarily  on 
instructional  quality  assessment  measures,  instructional  design  and  computer- 
based  delivery  pedagogical  guidelines. 

D.  LIMITATIONS 

In  its  Division  Officer  at  Sea  Training  Program  Overview,  IDSI  identifies 
seven  key  elements  that  comprise  the  SWO  DIVO  At  Sea  training  program. 
Those  elements  are  an  Individual  Development  Plan  (IDP),  Interactive 
Courseware  (ICW),  Practicums,  Practical  Problems,  Case  Studies,  Shipboard 
Experiences,  and  Assessment  (p.  3-1).  IDSI’s  overview  emphasizes  four 
instructional  categories,  namely  ICW,  Practicums,  Practical  Problems,  and  Case 
Studies,  which  are  important  to  the  successful  implementation  of  this  training 
endeavor.  However,  this  thesis  will  primarily  focus  on  the  curriculum  integration 
strategy,  and  in  particular,  a  critical  analysis  of  the  instructional  strategies  and 
delivery  methods  employed  to  create  the  SWO  division  officer  interactive 
courseware. 

E.  METHODOLOGY 

In  order  to  evaluate  whether  a  course  of  instruction  meets  its  objectives, 
one  must  be  aware  of  the  cognitive  and  pedagogical  principles  that  must  be 
satisfied  in  order  to  ensure  that  learning  occurs.  Consequently,  this  thesis  first 
examines  leading  cognitive  and  pedagogical  models  and  theories  of  learning.  A 
generic  evaluative  model  for  evaluating  computer-based  instructional  delivery 
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systems  is  developed  and  presented.  Although  developed  primarily  to  analyze 
computer-based  instructional  systems,  with  minor  modifications,  it  is  useful  in 
evaluating  traditional  educational  environments  as  well.  Next,  the  traditional 
(classroom)  Surface  Warfare  Officer  Curriculum  is  briefly  examined  and  then  the 
evaluation  template  is  used  to  thoroughly  evaluate  the  relative  strength  and 
weaknesses  of  the  reengineered  SWO  courseware  against  existing  cognitive  and 
pedagogical  principles.  Finally,  the  evaluation  results  are  summarized  and  all 
conclusions  and  recommendations  presented. 

Qualified  Surface  Warfare  Officers  serving  as  training  officers  aboard  their 
respective  ships  completed  surveys  developed  for  supervisors.  These  officers 
have  all  successfully  passed  the  Surface  Warfare  course  of  instruction  at 
Newport,  Rhode  Island,  and  having  mastered  the  skills  that  Ensigns  will  be 
required  to  learn,  have  earned  the  right  to  wear  the  Surface  Warfare  Pin. 
Ensigns  that  are  currently  enrolled  in  the  new  SWO  basic  course  completed 
surveys  that  were  devebped  for  students.  The  Ensigns  were  used  because  it  is 
important  to  discover  how  individuals  who  are  actually  taking  the  computer-based 
course  feel  about  the  way  it  is  being  delivered.  It  cannot  be  overemphasized  that 
if  the  students’  needs  are  not  being  met,  then  they  are  probably  not  learning  to 
the  best  of  their  potential,  and  consequently,  there  is  a  high  probability  that  the 
organization’s  training  objectives  are  not  being  met. 

F.  ORGANIZATION  OF  PAPER 

Chapter  II  examines  memory,  cognition,  and  a  review  of  the  major 
theories  of  learning.  Chapter  III  details  the  basics  of  a  typical  distance-learning 
program  and  develops  a  model  for  the  successful  evaluation  of  a  computer- 
based  course  of  instruction.  Chapter  IV  reviews  various  aspects  regarding  the 
traditional  SWO  classroom -based  course  of  instruction  and  evaluates  the 
cognitive  and  pedagogical  aspects  of  the  reengineered  SWO  CD-ROM  based 
courseware.  Chapter  V  presents  conclusions  and  recommendations  for 
improvements  to  this  new  course  of  instruction. 
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II.  MEMORY,  COGNITION,  AND  LEARNING 


Human  perception,  memory,  and  cognition  play  an  important  role  in  CBT 
programs.  To  effectively  develop  learning  materials  for  students,  course 
designers  must  take  the  way  people  learn  into  account.  To  varying  degrees, 
mankind  processes  information  similarly.  There  are,  of  course,  disputes  among 
theorists  about  how  this  process  works,  but  that  does  not  change  the  fact  that 
human  beings  sense,  process,  synthesize,  remember,  and  recall  information  in 
the  same  way.  Snce  psychology  has  such  a  large  impact  on  education  and 
training,  it  is  appropriate  to  include  information  about  memory,  cognition,  and 
human  learning  in  any  research  that  is  related  to  CBT. 

A.  MEMORY 

For  the  purposes  of  this  paper,  human  memory  is  the  ability  to  store, 
process,  retain,  and  recall  information  over  the  course  of  a  lifetime.  Memory  is 
tremendously  important.  Dix,  Finlay,  Abowd,  and  Beale  (1997)  demonstrate  that 
most  of  our  everyday  activities  rely  on  memory,  either  in  the  storage  of  actual 
facts,  or  the  knowledge  of  actions  or  procedures.  Our  memory  allows  us  to 
repeat  actions,  use  language,  and  utilize  new  information  that  is  received  via  our 
senses  (p.  26).  Humankind  is  endowed  with  the  ability  to  perceive  our 
environment.  This  perception  comes  as  a  result  of  taste,  smell,  sight,  hearing, 
and  touch,  which  are  the  five  senses. 

1. Senses 

Wickens  (1992)  states  each  one  of  the  five  human  sensory  systems  is 
equipped  with  a  central  mechanism  that  prolongs  the  representation  of  a  physical 
stimulus  for  a  short  period  of  time  after  the  stimulus  has  stopped  (p.  17).  This  is 
known  as  the  Short  Term  Sensory  Store  (STSS).  Additionally,  Wickens  (1992) 
and  Dix  et  al  (1997)  contend  that  when  a  person’s  attention  is  diverted,  the  STSS 
acts  as  a  “stimuli”  buffer,  and  permits  environmental  information  to  be  temporarily 
preserved  so  that  it  can  be  dealt  with  later.  According  to  Dix  et  al.  (1997),  a 
STSS  exists  for  each  sensory  channel:  iconic  memory  for  visual  stimuli,  echoic 
memory  for  aural  stimuli,  and  haptic  memory  for  touch  p.  1 7). 
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Wickens  (1 992)  relates  three  general  characteristics  of  the  STSS: 

(1)  The  STSS  is  preattentive.  This  means  no  conscious  attention  is  required  to 
prolong  the  image  during  the  natural  “time  constant”  of  the  sensory  store  (p.  1 8). 

(2)  The  STSS  is  relatively  veridical,  preserving  most  of  the  physical  details  of  the 
stimulus  (p.  18). 

(3)  The  information  in  the  STSS  decays  rapidly.  The  iconic  store  holds  sensory 
data  for  less  than  one  second.  The  echoic  and  haptic  stores  generally  hold 
information  between  two  and  eight  seconds  (p.  1 8). 

2.  Short-Term  Memory 

Information  that  is  resident  within  a  STSS  must  be  acted  upon  if  it  is  to  be 
remembered.  Anderson  (2000)  shows  information  that  was  left  “unattended”  in  a 
STSS  was  quickly  lost.  Attended,  or  acted  upon  information,  went  into  short-term 
memory  (p.  172).  Short-term  memory  has  become  synonymous  with  working 
memory.  Anderson  (2000)  and  Breuer  (1990)  also  demonstrated  if  short-term 
memory  was  not  acted  upon,  or  rehearsed,  it  was  quickly  forgotten  as  well.  Dix 
et  al.  (1997)  believe  working  memory  acts  as  a  “scratchpad”  for  the  temporary 
recall  of  information  (p.  28). 

Short-term  memory  has  a  limited  capacity.  Anderson  (2000)  reports  the 
number  of  elements  in  a  sequence  that  a  person  can  ordinarily  repeat  back 
without  error  is  between  seven  or  eight  (p.  17).  For  example,  a  person  can 
usually  remember  a  seven-digit  phone  number  quite  easily.  However,  if  that 
phone  number  includes  an  area  code,  bringing  the  number  of  digits  to  ten,  it 
often  becomes  necessary  to  write  the  number  on  a  piece  of  paper  and  rehearse 
it  before  it  is  recalled  correctly.  Anderson  (2000)  believes  the  number  of 
rehearsals  control  the  amount  of  information  that  is  transferred  to  long-term 
memory  (p.  17).  In  fact,  Anderson  (2000)  argues  that  study  time  and  the  amount 
of  practice  with  a  given  data  set  directly  contributes  to  a  person’s  ability  to 
accurately  recall  that  data  when  it  is  needed.  Figure  1  shows  the  relationship 
between  senses,  short-term  memory,  and  long-term  memory. 
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Figure  1 .  Sensory  to  Long-term  memory  (From  Dix  et  al.,  1 997,  p.  27) 

CBT  design  should  take  into  account  the  need  to  act  on  information 
multiple  times  so  it  can  be  sent  from  STSS  through  short-term  memory  to  long¬ 
term  memory. 

3.  Long-Term  Memory 

Anderson  (2000),  Wickens  (1992),  and  Dix  et  al.  (1997)  show  sensory 
information  is  acted  upon  and  processed  by  progressively  higher  neural  centers, 
and  as  it  continues  to  be  acted  upon,  it  will  eventually  be  perceived  or 
recognized,  and  then  stored  in  long-term  memory.  Dix  et  al.  (1997)  contend 
long-term  memory  is  a  human  beings  main  resource  for  storing  factual 
information,  experimental  knowledge,  and  procedural  rules  of  behavior  (p.  30). 
Dix  et  al.  (1997)  discuss  two  types  of  long-term  memory.  There  is  episodic 
memory,  which  is  a  memory  of  events  and  experiences  in  a  serial  form.  There  is 
also  semantic  memory,  which  is  structured  to  allow  access  to  information, 
representation  of  relationships  between  pieces  of  information,  and  inference  (p. 
31).  Long-term  memory  differs  from  short-term  memory  in  two  very  distinct 
ways.  First,  Long-term  memory  has  a  nearly  infinite  capacity.  Second,  forgetting 
happens  at  a  much  slower  rate. 

4.  Semantic  Memory  Maps 

Anderson  (2000)  states  that  in  one’s  semantic  memory,  information  is 
broken  down  categorically.  Thus,  structure  is  represented  as  a  hierarchy  of 
categorized  facts.  Categorized  facts  are  then  associated  with  the  different 
categories  (p.  152).  A  semantic  network  is  a  series  of  interconnected  semantic 
maps.  Figure  2  shows  a  representation  of  a  semantic  memory  map. 
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Schemas  are  developed  from  semantic  networks.  Anderson  (2000) 
contends  that  since  schemas  are  built  from  semantic  networks,  the  sum  of  our 
knowledge  about  a  semantic  category  can  be  captured  (p.  154).  Anderson 
(2000)  also  argues  that  categories  have  attributes,  and  that  many  of  those 
attributes  have  a  “default”  value  associated  with  them.  Furthermore,  if  somebody 
recognizes  an  object  as  belonging  to  a  certain  category,  that  person  can  infer 
that  the  object  has  the  default  values  associated  with  that  concept’s  schema  (p. 
1 54).  Figure  2  is  an  example  of  a  semantic  map. 
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Figure  2.  A  semantic  mapping  of  ships  (After  Anderson’s  (2000)  model  (p.  1 52)) 

5.  Memory  Relevance 

It  is  very  important  that  curriculum  developers  understand  the  different 
aspects  of  human  memory  processes  so  CBT  presentations  can  be  constructed 
that  take  full  advantage  of  the  way  people  learn,  and  maximize  the  educational 
benefit  to  students.  If  a  series  of  CBT  lessons  are  related,  and  the  ways  in  which 
they  are  related  are  incorporated  into  the  presentations,  the  students  are  in  fact 
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given  the  means  to  make  mental  connections  between  the  various  information 
groups  and  categories.  Consequently,  it  is  possible  to  use  CBT  to  improve  the 
way  students  construct  semantic  maps,  ultimately  improving  memory  and 
performance. 

B.  KNOWLEDGE  APPLICATION  AND  REASON 

Anderson  (2000)  describes  three  ways  humans  acquire  knowledge. 
These  are  by  personal  discovery,  through  instruction  (being  told),  and  by 
observing  somebody  else’s  performance  (p.  246). 

1.  Knowledge 

There  are  two  types  of  knowledge.  Anderson  (2000)  states  these  are 
declarative  knowledge  and  procedural  knowledge.  Declarative  knowledge  is  also 
known  as  explicit  knowledge,  and  is  the  knowledge  we  are  consciously  aware  of, 
including  facts,  dates,  events,  etc.  Procedural  knowledge  is  the  knowledge  of 
how  to  perform  cognitive  activities.  This  type  of  knowledge  is  often  implicit  (p. 
238). 

Knowledge  acquisition,  or  learning,  can  be  classified  into  two  categories: 
tactical  learning  and  strategic  learning.  As  students  practice  problems  they  learn 
the  sequences  of  actions  required  to  solve  problems  or  portions  of  problems. 
Anderson  (2000)  classifies  this  as  tactical  learning  (p.  290).  Students  also  learn 
how  to  organize  problem-solving  techniques,  particularly  as  problems  get  large 
and  complex.  Anderson  (2000)  labels  this  as  strategic  learning  (p.  292).  The 
more  information  students  learn  about  a  subject,  the  more  it  is  assumed  they 
understand  about  that  subject.  Wiggins  (1998)  defines  understanding  as  a 
sufficient  grasp  of  concepts,  principles,  or  skills  so  that  one  may  bring  them  to 
bear  on  new  problems  and  situations,  deciding  in  which  ways  one’s  present 
competencies  can  suffice  and  in  which  ways  one  may  require  news  skills  or 
knowledge  (p.  84).  Student  understanding,  then,  must  be  a  major  goal  of  any 
educational  endeavor. 

2.  Reasoning 

Reasoning  is  the  application  of  knowledge  to  solve  problems  or  perform 
tasks.  Dix  et  al.  (1997)  contend  that  reasoning  occurs  in  three  distinct  ways: 
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deductively,  inductively,  and  through  abduction.  Deductive  reasoning  means 
conclusions  are  logically  derived  from  a  set  of  given  or  known  premises. 
However,  a  logical  conclusion  cannot  be  equated  to  “truth”.  Just  because 
something  should  logically  be  a  certain  way,  does  not  mean  it  is  really  that  way. 
When  using  inductive  reasoning,  a  person  generalizes  from  a  set  of  known  facts 
to  infer  information  about  cases  that  have  not  >et  been  seen.  Using  abductive 
logic,  a  person  would  step  logically  from  a  given  fact  to  the  action  or  state  that 
caused  it  (pp.  38-40). 

3.  Expertise 

Over  time,  and  through  the  use  of  learning  strategies,  reasoning,  and 
problem  solving,  a  person  may  become  known  as  an  expert  about  a  given 
subject.  Anderson  (2000)  describes  three  stages  of  skill  acquisition  and 
expertise.  These  stages  are  the  cognitive  stage,  the  associative  stage,  and  the 
autonomous  stage.  In  the  cognitive  stage,  sets  of  relevant  facts  are  committed  to 
memory.  In  the  associative  stage,  two  actions  occur.  First,  errors  in  the  initial 
understanding  are  gradually  detected  and  eliminated.  Second,  the  connections 
among  the  various  elements  required  for  successful  performance  are 
strengthened  (p.  281).  The  outcome  from  this  stage  is  the  internalization  of  a 
successful  procedure  for  the  performance  of  a  given  skill  set.  In  the  autonomous 
stage,  Anderson  (2000)  states  that  procedural  knowledge  produced  in  the 
associative  stage  becomes  more  and  more  automated  and  rapid.  Since  skills 
are  becoming  more  automatic,  they  require  fewer  processing  resources  (p.  282). 
For  a  given  skill  set,  a  person  operating  in  this  stage  will  generally  be  accepted 
as  a  subject  matter  expert  (SME). 

C.  TRANSFER 

Allessi  and  Trollip  (1991)  and  Ellis  (1965)  define  transfer  as  skills  or 
performance  in  one  situation  influencing  behavior  in  other  situations.  Transfer  is 
also  commonly  used  to  describe  taking  information  learned  in  an  educational 
environment  and  applying  it  to  situations  that  occur  outside  of  the  classroom. 
Ellis  (1965)  reports  that  the  influence  of  transfer  is  found  in  intellectual  tasks, 
complex  motor  skills,  and  in  emotional  responses  and  attitudes  (p.  3).  Three 
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types  of  transfer  can  occur  as  a  result  of  instruction.  They  are  positive  transfer, 
negative  transfer,  and  zero  transfer. 

Positive  transfer  can  be  defined  as  the  performance  of  one  task  aiding  in 
the  performance  of  a  second  task  (Ellis,  1995,  p.  4).  Ellis  (1995)  argues  the  act 
of  driving  a  car  can  be  used  to  explain  transfer.  Assume  a  teenager  has  learned 
to  drive  the  family  car.  Assuming  a  similar  transmission,  that  teenager  should 
have  little  difficulty  driving  the  neighbor’s  car  (p.  4). 

Ellis  (1965)  defines  negative  transfer  as  the  performance  on  one  task 
inhibiting  or  disrupting  the  performance  on  a  second  task  (p.  4).  Ellis  (1995) 
again  uses  the  act  of  driving  a  car  as  an  example.  Learning  to  drive  in  the  United 
States  is  different  than  learning  to  drive  in  Great  Britain.  A  person  that  has  lived 
and  driven  in  the  United  States  for  a  prolonged  period  of  time  would  more  than 
likely  find  it  very  difficult  to  drive  in  Great  Britain,  on  the  left  side  of  the  road  (p.  5). 
This  is  an  example  of  negative  transfer. 

Zero  transfer  means  that  the  performance  of  one  task  has  no  effect  on  the 
performance  of  another. 

There  are  many  factors  that  affect  the  degree  of  transfer  that  occurs 
between  one  task  and  another.  Ellis  (1965)  contends  that  task  similarity, 
response  similarity,  and  the  time  between  tasks  all  influence  how  much  transfer 
occurs. 

In  general,  increased  task  similarity  results  in  increased  positive  transfer 
unless  the  stimulus  is  the  similar  but  the  responses  are  different.  Ellis  (1965) 
explains  this  phenomenon  by  demonstrating  a  motorist’s  behavior  while  at  a 
traffic  signal.  Consider  changing  the  rules  to  “go”  on  orange  instead  of  going  on 
green.  Since  the  color  orange  is  similar  to  red,  which  means  stop,  motorists 
would  have  problems  adapting  to  the  new  rules.  However,  if  the  green  lights 
were  replaced  with  blue  ones,  and  then  the  rule  was  changed  to  go  on  blue,  it 
would  take  little  time  before  all  the  motorists  had  adapted  to  the  change. 
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When  a  person  must  make  new  responses  to  similar  stimuli,  negative 
transfer  usually  results.  Ellis  (1965)  uses  a  person  learning  another  language  as 
an  example.  Consider  the  person  fluent  in  English  who  is  attempting  to  learn 
German.  Now  consider  the  word,  “wand”.  In  English,  a  wand  is  the  Fairy 
Godmother’s  tool.  In  German,  it  means  a  wall.  Ellis’  research  shows  that  the 
person  will  have  a  difficult  time  remembering  and  using  the  German  definition 
properly. 

It  was  reported  by  Ellis  (1965)  that  transfer  remains  constant,  even  with 
intervals  that  vary  between  zero  and  ninety  days  between  tasks  (p.  39).  Transfer 
of  learning  appears  to  be  stable  and  independent  of  any  memory  of  the  original 
tasks. 

Bastiaens  and  Martens  (2000)  contend  that  teaching  students  outside  the 
context  of  the  actual  work  environment  results  in  a  low  transfer  of  knowledge  and 
skill  (p.  8).  Not  surprisingly,  Bastiaens  and  Martens  (2000)  advocate  the  use  of 
on-the-job  training  (OJT)  for  organizational  training.  It  is  possible  that  OJT  has  a 
few  advantages  over  the  traditional  classroom  approach.  Bastiaens  and  Martens 
(2000)  argue  that  in  an  OJT  environment,  the  learning  environment  and  the  job 
location  are  the  same,  and  positive  transfer  will  increase  (p.  8).  It  is  also  argued 
that  since  the  learning  process  in  an  OJT  environment  is  more  active  and 
concrete  than  in  the  classroom,  transfer  will  improve  (Bastiaens  and  Martens, 

2000,  p.  8). 

While  OJT  does  possess  educational  benefits,  it  is  also  likely  that  learning 
and  transfer  from  this  form  of  training  are  at  times  reduced.  Transfer  also  occurs 
in  the  affective  domain.  The  affective  domain  is  primarily  constructed  of  attitudes 
and  opinions.  A  person  that  thinks  poorly  of  his  or  her  place  of  employment  may 
possess  negative  feelings  toward  any  OJT.  It  is  also  possible  that  the  training 
day  is  often  interrupted,  which  will  diminish  the  impact  of  OJT. 

D.  LEARNING  PRINCIPLES 

The  subject  of  learning  is  both  broad  and  extremely  complex.  In  order  to 
construct  a  useful  CBT  program,  it  is  important  to  have  an  elementary 
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understanding  of  how  students  learn  and  what  instructors,  even  if  the  instructor  is 
a  computer,  can  do  to  enhance  the  learning  process.  Understanding  the  three 
learning  domains  that  exist  in  each  person,  the  concept  of  metacognition,  and  the 
four  human  factors  that  affect  learning  are  very  important.  By  understanding  how 
these  principles  work  together,  curriculum  developers  can  construct  CBT 
software  that  will  allow  students  to  maximize  their  learning  potential. 

1.  Krathwohl,  Bloom,  and  Bertram’s  Domains 

Krathwohl,  Bloom,  and  Bertram  (1964)  define  three  domains  in  which 
learning  occurs.  These  are  the  cognitive,  affective,  and  psychomotor  domains. 
These  domains  are  defined  in  terms  of  intellectual,  mental,  or  physical  objectives. 
Depending  upon  what  a  organization’s  particular  training  objectives  are,  activities 
can  be  developed  that  stimulate  student  learning  across  each  of  these  domains. 
If,  however,  an  organization  has  poorly  defined  learning  objectives,  it  makes  it 
harder  to  develop  specific  training  activities  that  will  both  meet  the  objectives  and 
enhance  learning  and  transfer  throughout  these  three  domains. 

a.  Cognitive  Domain 

This  is  the  intellectual  part  of  a  person.  This  is  the  domain  that  is 
responsible  for  knowledge  acquisition,  recall,  and  problem  solving.  Krathwohl, 
Bloom,  and  Bertram  (1964)  define  the  cognitive  domain  as  objectives  which 
emphasize  remembering  or  reproducing  something  which  has  presumably  been 
learned,  as  well  as  objectives  which  involve  the  solving  of  some  intellective  task 
for  which  the  individual  has  to  determine  the  essential  problem  and  then  reorder 
given  material  or  combine  it  with  ideas,  methods,  or  procedures  previously 
learned  (p.  6). 

b.  Affective  Domain 

Krathwohl,  Bloom,  and  Bertram  (1964)  define  the  affective  domain 
as  objectives  that  emphasize  a  feeling,  tone,  emotion,  or  a  degree  of  acceptance 
or  rejection.  This  is  simply  a  person’s  attitudes,  values,  feelings,  emotions,  and 
biases  (p.  6). 
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c.  Psychomotor  Domain 

Krathwohl,  Bloom,  and  Bertram  (1964)  define  the  psychomotor 
domain  as  objectives  that  emphasize  some  muscular  or  motor  skill,  some 
manipulation  of  material  and  objects,  or  some  act  that  requires  neuromuscular 
coordination,  such  as  writing,  speech,  or  a  trade  (p.  7). 

2.  Metacognition 

Stolovitch  (2003)  defines  metacognition  as  the  set  of  higher-level  control 
processes  that  guide  our  deliberate  information  processing  activities.  It  is  in 
essence,  the  mind’s  “Operating  System”  (OS)  (p.  88).  Stolovitch  (2003) 
identifies  five  such  processes.  They  are  planning,  selecting,  connecting,  tuning, 
and  monitoring  (p.  90). 

These  processes  are  bipolar.  On  one  end  are  characteristics  that  a  good 
learner  typically  exhibits.  On  the  opposite  pole  are  the  characteristics  that  a  poor 
learner  generally  displays.  Positions  on  this  scale  are  not  static,  but  dynamic. 
Over  time,  a  person  can  improve  their  performance  within  one  or  all  of  these 
processes’  spectrums  with  practice. 

a.  Planning 

The  process  of  planning  ranges  from  a  poor  learner  not  knowing 
what  to  do  to  a  good  learner  reasoning  out  what  must  be  done  by  creating  and 
executing  an  organized  plan  of  action  (Stolovitch,  2003,  p.  90). 

b.  Selecting 

The  process  of  selecting  ranges  from  a  poor  learner  not  knowing 
where  to  turn  or  select  what  is  important  to  learn  to  the  good  learner  listening, 
studying,  analyzing,  and  sifting  through  information,  identifying  and  selecting  the 
critical  elements  (Stolovitch,  2003,  p.  90).  In  other  words,  the  good  students  are 
able  to  figure  out  what  is  important  to  learn. 

c.  Connecting 

Stolovitch  (2003)  contends  that  those  who  do  not  connect  well  view 
new  content  as  a  mass  to  be  analyzed  and  will  attempt  to  memorize  that  content 
without  linkages  to  known  skills  and  knowledge.  Those  who  do  connect  well 
continually  seek  to  build  linkages  with  their  prior  knowledge  (p.  90). 


14 


d.  Tuning 

Stolovitch  (2003)  argues  that  those  that  are  poor  at  tuning  will  only 
obtain  a  fuzzy  understanding  of  new  knowledge  and  will  be  unable  to  pull  that 
knowledge  into  focus.  On  the  other  hand,  good  students  intentionally  practice 
with  new  information  until  it  is  brought  into  a  clear  and  sharp  focus  (p.  90). 

e.  Monitoring 

Students  with  poor  monitoring  skills  use  known  learning  strategies 
whether  those  strategies  work  or  not.  These  students  tend  to  apply  more  effort 
into  one  strategy  if  a  learning  outcome  is  negative,  or  unproductive,  than  to  apply 
a  different  strategy.  Students  who  can  effectively  monitor  replace  unproductive 
or  insufficient  strategies  with  ones  more  likely  to  work  (Stolovitch,  2003,  p.  90). 

Stolovitch  (2003)  believes  that  teachers  can  help  students  of  all 
ages  improve  in  each  of  these  areas  by  designing  learning  activities  that  guide 
students  in  the  application  of  these  skills,  which  forces  them  to  actively  think 
about  and  evaluate  what  they  are  doing  and  why  they  are  doing  it  (p.  88). 

3.  Factors  that  Affect  Learning 

There  are  four  factors  that  have  an  impact  on  the  way  a  student  learns. 
These  factors  are:  ability,  prior  knowledge,  motivation,  and  thinking  style. 

a.  Ability 

Stolovich  (2003)  defines  ability  as  the  capacity  with  which  we  were 
born  that  enables  us  to  acquire  new  skills  and  knowledge  (p.  36).  This  is  also 
known  as  intellectual  capacity,  and  it  varies  from  person  to  person.  Those  with  a 
high  intellectual  capacity  will  typically  understand  and  recall  information  better 
than  those  without  such  a  capacity. 

b.  Prior  Knowledge 

Prior  knowledge  is  defined  as  how  much  a  person  already  knows 
about  what  is  being  currently  taught.  Stolovitch  (2003)  states  that  prior 
knowledge  helps  the  learner  acquire  additional  knowledge  or  skills  more  quickly 
(p.  37). 
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c.  Motivation 

Motivation  can  be  defined  several  ways.  I  prefer  to  define 
motivation  as  a  person’s  desire  to  achieve  a  stated  purpose  or  goal.  Stolovitch 
(2003)  believes  that  motivation  is  affected  by  three  factors:  value,  confidence, 
and  mood  (p.  37).  Stolovitch  (2003)  argues,  and  it  makes  sense,  the  more  a 
person  values  something;  the  more  motivated  that  person  will  be  with  respect  to 
what  is  valued.  In  terms  of  learning,  this  means  that  the  more  value  a  person 
affixes  to  learning,  the  more  motivated  that  person  will  be  to  learn. 

Additionally,  Stolovitch  (2003)  argues  that  confidence  plays  a  large 
part  in  a  person’s  motivation.  If  a  person  feels  completely  inept  at  something  it  is 
unlikely  they  will  be  motivated  to  do  or  try  it  (p.  38). 

Stolovitch  (2003)  believe  that  a  person’s  personal  feelings  affect 
their  moods,  and  that  those  moods  affect  motivation  (p.  38).  A  positive 
environment  tends  to  improve  one’s  mood,  and  consequently,  improve  motivation 
(Stolovitch,  2003,  p.  38). 

There  are  two  forms  of  motivation.  Allessi  and  Trollip  (1991) 
describe  them  as  intrinsic  motivation  and  extrinsic  motivation.  Intrinsic 
motivators  are  things  that  are  inherent  in  the  instruction  that  motivate  a  student 
(p.  31).  Allessi  and  Trollip  (1991)  describe  exploratory  environments,  challenging 
assignments,  tasks  that  stimulate  curiosity,  and  encouragement  as  methods  that 
increase  a  student’s  intrinsic  motivation  (p.  31).  Allessi  and  Trollip  (1991), 
describe  extrinsic  motivators  as  those  that  are  independent  of  the  instruction  (p. 
31).  Paying  a  student  for  good  grades  is  an  example  of  an  extrinsic  motivator. 
Allessi  and  Trollip  (1991)  argue  that  when  extrinsic  motivators  are  used,  the 
student’s  interest  is  diminished  because  the  goal  becomes  the  reward  instead  of 
learning  (p.  31). 

1.  Learning  Environment  and  Motivation.  The  learning 
environment  can  have  a  large  impact  on  a  student’s  motivation.  There  are  four 
different  learning  environments.  Kirkpatrick  (1998)  describes  them  as 
discouraging,  neutral,  encouraging,  and  requiring.  In  a  discouraging 
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environment,  the  supervisor  often  says,  “I  want  it  done  this  way”,  irrespective  of 
the  actual  organizational  rules.  A  person  is  not  often  told  directly  not  to  do 
something,  but  behavior  is  regulated  through  insinuation.  According  to  Allessi 
and  Tollip  (1991),  a  neutral  environment  is  one  in  which  the  boss  ignores  the  fact 
that  trainees  have  attended  training  (p.  21).  In  effect,  the  boss  is  saying,  “I  do  not 
care  what  you  have  learned,  do  it  this  way.”  In  an  encouraging  environment,  the 
organization  assists  the  student  in  transferring  what  was  learned  in  training  to  the 
job.  The  “requiring”  environment  is  the  most  difficult  to  implement.  In  this 
environment,  the  supervisor  knows  what  the  subordinate  learns  and  then 
ensures  that  the  students’  learning  transfers  to  the  job  (Allessi  and  Trollip,  1991, 
P-21). 

Based  upon  these  descriptions,  organizations  should  strive 
to  implement  either  encouraging  or  requiring  environments.  By  doing  so,  more 
transfer  will  take  place  between  training  and  the  job,  trainee  satisfaction  and 
motivation  will  increase,  and  the  organization  will  benefit  from  a  better-trained, 
more  responsive  workforce. 

d.  Thinking  Styles 

Sternberg  (1997)  argues  that  in  addition  to  ability,  prior  learning, 
and  motivation,  thinking  styles  significantly  impact  a  person’s  ability  to  learn. 
Sternberg  (1997)  identifies  three  distinct  thinking  styles.  They  are  legislative, 
executive,  and  judicial.  Legislative  thinkers  like  to  come  up  with  their  own  ways 
of  doing  things.  They  also  prefer  to  decide  for  themselves  what  they  will  do  and 
how  they  will  do  it  by  creating  their  own  rules.  Legislative  thinkers  tend  to  make 
good  writers,  scientists,  artists,  and  bankers  (Sternberg,  1997,  pp  20-21). 

Sternberg  (1997)  states  that  executive  thinkers  like  to  follow  rules 
and  prefer  problems  that  are  prefabricated.  They  like  to  fill  in  gaps  that  exist 
within  existing  structures  rather  than  create  the  structures  themselves.  They  also 
like  applying  rules  to  problems.  These  types  of  thinkers  make  good  lawyers, 
police  officers,  contractors,  and  soldiers  (p.  21). 
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Judicial  thinkers  like  to  evaluate  rules  and  procedures,  preferring 
problems  in  which  one  analyzes  and  evaluates  existing  constructs  and  ideas 
(Sternberg,  1997,  p.  21).  Sternberg  (1997)  argues  that  these  people  like  writing 
critiques,  giving  opinions,  and  judging  people  and  their  work.  Judicial  thinkers 
make  good  judges,  critics,  program  evaluators,  consultants,  and  systems 
analysts  (p.  21). 

Since  people  all  operate  using  different  thinking  styles,  questions 
placed  on  a  test  will  usually  result  in  different  answers,  depending  on  the  thinking 
style  of  the  person  asked.  It  is  important  to  keep  each  of  these  styles  in  mind 
when  constructing  tests  and  activities  with  which  to  evaluate  students.  This  is 
because  a  test  constructed  one  way  may  favor  a  particular  thinking  style  over 
another.  Asking  for  the  same  information  in  different  ways  or  constructing 
multiple  activities  that  demonstrate  the  same  skill  are  methods  for  eliminating  any 
bias  that  may  exist  with  respect  to  the  students’  thinking  styles. 

E.  APPROACHES  TO  LEARNING 

There  are  many  theories  about  how  to  best  teach  students.  The  three  that 
will  be  discussed  in  this  research  are  the  behaviorist  theory,  the  constructivist 
theory,  and  the  cognitive  theory.  Some  theories  have  more  relevance  in  a 
traditional  classroom,  while  some  seem  to  fit  best  within  the  confines  of  some 
computer-based  environments.  Each  theory  has  both  advantages  and 
disadvantages  with  respect  to  the  learning  environment  in  which  it  is  applied. 
The  goal  of  this  research  with  respect  to  learning  theories  is  to  highlight  the  basic 
similarities  and  differences  between  the  theories  and  to  briefly  discuss  possible 
strengths  or  weaknesses  of  the  three  theories  in  the  context  of  organizational 
training  via  CBT. 

1.  Behaviorism 

Smith-Grato  (2000)  traces  the  roots  of  behaviorism  to  Skinner’s  Theory  of 
Programmed  Instruction  (p.  228).  Programmed  instruction  was  founded  on 
learning  principles  largely  determined  by  animal  learning  studies. 

Kidd  (1965)  believes  that  learning  is  primarily  the  alteration  of  behavior 

that  results  from  experience  (p.  154).  Kidd  (1965)  is  saying  that  we  learn  from 
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what  we  do,  not  from  what  we  accomplish  (p.  154).  When  I  was  a  young  boy,  I 
touched  a  hot  burner  on  a  stove.  Having  learned  from  my  experience,  I  have 
never  again  touched  a  stove  burner  that  may  even  have  been  hot. 

Kidd  (1965)  describes  the  principle  known  as  the  “curves  of  practice”. 
This  principle  demonstrates  that  for  a  given  activity,  the  gains  in  learning  or 
expertise  increases  the  more  that  activity  is  practiced  (p.  156).  Consider  a 
person  learning  to  play  the  guitar.  The  more  a  person  practices  playing  the 
guitar,  the  better  he  or  she  will  become. 

Desypris  (2002)  lists  three  steps  that  behaviorists  believe  need  to  be 
performed  in  order  to  effectively  teach.  They  are: 

(1)  Instructors  must  identify  the  objectives  to  be  learned. 

(2)  Instructors  must  create  a  learning  environment  that  assists  the  learner  in 
acquiring  these  goals.  This  environment  includes  stimulus  that  will  engage  the 
learner. 

(3)  Instructors  must  review,  examine,  and  consider  adopting  existing  materials 
into  their  curriculums  before  developing  new  ones  (p.  1 8). 

Behaviorism  was  a  very  popular  theory  in  the  1950’s  and  1960’s. 
Behavioral  learning  principles  tended  to  emphasize  simple,  incremented  steps 
representing  a  particular  task  domain.  The  behavioral  method  of  programmed 
instruction  has  been  successfully  applied  in  limited  training  environments  on 
tasks  that  require  performance  of  simple  procedures.  Even  though  this  form  of 
instruction  largely  ignores  human  problem  solving  abilities  and  higher  cognitive 
skills,  it  still  provides  a  nice  foundation  on  which  to  build. 

2.  Cognitive  Theory 

Duffy  and  Jonassen  (1992)  describe  cognitive  science  as  multidisciplinary, 
drawing  on  psychology,  linguistics,  anthropology,  philosophy,  and  artificial 
intelligence  (p.  20).  Duffy  and  Jonassen  (1992)  argue  that  the  mind  is  essentially 
viewed  as  an  instantiation  of  a  computer,  manipulating  symbols  the  same  way 
that  a  computer  does.  These  “symbols”  acquire  meaning  when  an  external  and 
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independent  realty  is  “mapped”  on  them  as  during  our  interaction  with  the  world. 
Cognition  is  the  rule-based  manipulation  of  these  symbols  (p.  20).  Leflore  (2000) 
states  that  individuals  develop  maps  and  schemas  to  help  them  understand  the 
world.  Individuals  will  reorganize  existing  concepts  when  those  concepts  interact 
with  new  experiences  (p.  105). 

Because  of  this,  Desypris  (2002)  argues  that  the  course  must  include 
problems  that  the  student  will  solve  in  step,  taking  advantage  of  acquired 
knowledge  in  each  step  (p.  18).  Additionally,  Desypris  (2002)  contends  that  in 
order  for  instruction  to  be  effective,  it  is  the  teacher’s  responsibility  to  ensure  that 
each  student  is  ready  to  learn  the  core  concepts  that  will  be  taught. 

Leflore  (2000)  provides  four  basic  guidelines  that  instructors  can  utilize 
when  developing  course  content: 

(1)  Provide  students  with  elements  that  help  them  structure  and  organize  the 
information  they  are  expected  to  learn.  Providing  students  with  an  outline  of  the 
material  they  are  going  to  learn  is  an  excellent  method  for  accomplishing  this 
task  (p.  105). 

(2)  When  appropriate,  use  a  concept  development  activity  (p.  105).  An  example 
of  this  would  be  to  ask  aspiring  students  in  a  nautical  navigation  class  to  draw  the 
light  configuration  of  a  particular  ship,  or  class  of  ships. 

(3)  Decide  how  students’  prior  knowledge  will  be  activated  (p.  105). 

(4)  Use  graphics,  animations,  and  sounds  that  are  related  to  the  content  being 
taught  (p.  105). 

3.  Constructivism 

The  theory  of  constructivism  has  gained  in  popularity  as  the  Internet,  and 
distance  learning  has  gained  in  popularity.  Duffy  and  Jonassen  (1992),  Leflore 
(2000),  and  Smith-Gratto  (2000)  contend  what  makes  this  theory  unique  is  its 
tenet  that  the  students  construct  their  own  meaning,  or  schemas,  based  on  their 
unique  individual  experiences.  Duffy  and  Jonassen  (1992)  argue  learning  is  an 
active  process,  built  on  the  internal  representations  of  knowledge,  which  has 
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been  constructed  from  personal  interpretations  of  one’s  experience.  This 
knowledge  is  constantly  open  to  change  and  its  structures  and  linkages  form  the 
foundation  to  which  knowledge  structures  are  appended  (p.  21).  One  way  of 
interpreting  this  viewpoint  is  through  the  semantic  mapping  model.  When  a 
person  is  young,  the  structure  of  their  semantic  map  is  limited  and  small.  Over 
time,  as  this  person  experiences  life,  those  mappings  become  large,  complex, 
and  interconnected. 

Duffy  and  Jonassen  (1992)  and  Leflore  (2000)  contend  that  for  any 
meaningful  learning  to  occur,  it  must  be  situated  in  a  rich  context  and  filled  with 
real-world  problems  for  students  to  solve.  Leflore  (2000)  states  simulations 
provide  enough  fidelity  for  students  to  explore  real-world  problems  from  multiple 
vantage  points  and  will  enable  students  to  “construct”  good  problem  solving  skills 
(P-112). 

According  to  Desypris  (2002),  this  theory  puts  students  in  charge  of  their 
own  learning.  Consequently,  course  content  must  be  well  structured  in  order  to 
provide  continuity  of  learning  to  the  students.  Since  experience  plays  such  a 
large  role  in  this  theory,  interaction  among  participants  is  a  very  important 
element  in  the  construction  of  structured  knowledge. 

Smith-Gratto  (2000)  describes  three  potential  problems  that  can  occur 
when  teaching  using  this  method: 

(1)  Students  may  not  meet  the  required  objectives  because  facts,  skills,  and 
concepts  may  not  have  been  learned.  This  can  happen  if  students  are  required 
to  make  meaning  from  unfamiliar  content.  If  students’  experiences  do  not 
overlap  with  new  information,  it  is  unlikely  that  the  concept  will  be  successfully 
learned  (p.  233). 

(2)  It  is  possible  that  students  will  not  construct  a  proper  foundation  on  which  to 
build  (p.  233). 

(3)  Constructivist  activities  are  very  time  consuming  (p.  233). 
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4.  Conclusion 

By  understanding  how  the  human  mind  receives,  stores,  processes,  and 
retrieves  information,  CBT  developers  can  design  courseware  solutions  that  give 
students  the  best  opportunity  to  maximize  their  learning  potential.  On  the  other 
hand,  it  is  likely  that  CBT  courseware  that  is  neglectful  of  how  we  learn  will  fail  to 
meet  the  organizational  objectives  for  which  it  was  designed. 

Although  these  three  theories  have  been  presented  independently,  there 
is  nothing  to  prevent  curriculum  developers  to  combining  ideas  and  activities 
from  each  one  in  order  to  provide  an  optimum  learning  environment  for  their 
students.  By  doing  so,  developers  can  maximize  the  benefit  offered  by  a 
particular  theory  and  minimize  the  associated  weaknesses  that  are  also  inherent. 
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III.  COMPUTER-BASED  LEARNING 


For  any  educational  initiative  to  be  successful,  whether  it  is  teaching 
English  to  fifth-graders  or  training  potential  pilots  how  to  master  the  mechanics  of 
a  particular  aircraft,  two  things  must  occur.  First,  the  curriculum  must  be  sound. 
This  means,  among  other  things,  the  information  being  presented  must  be 
accurate,  comprehensive,  relevant,  and  timely.  Second,  the  information  must  be 
effectively  communicated,  or  delivered,  to  the  students.  Using  appropriate 
Instructional  Design  techniques  generally  allows  for  the  development  of  a  sound 
curriculum.  Proper  pedagogical  principles  must  be  adhered  to  in  order  to 
produce  quality  instructional  presentations. 

The  quality  of  instructional  design  and  delivery  techniques  is  assessed 
through  the  use  of  evaluations.  Evaluations  allow  organizations  to  determine  if 
training,  and  the  performance  goals  that  arise  out  of  the  completion  of  the 
training,  are  being  met  or  exceeded.  The  evaluation  framework  that  is  presented 
at  the  end  of  this  chapter  was  constructed  by  researching  instructional  design 
and  pedagogical  principles  that  have  been  proven  to  be  effective  in  a  CBT 
environment  and  then  building  a  template  that  checks  to  see  if  those 
characteristics  are  present  in  the  course  of  instruction  being  evaluated. 

A.  INSTRUCTIONAL  DESIGN 

Instructional  Design  can  be  defined  as,  “The  process  of  designing  the 
environment,  methods,  and  resources  for  effective  learning  and  objectives  by 
students  (Boettcher  and  Conrad,  1999,  p.  42).  As  quoted  by  Desypris  (2002), 
Hiltz,  Teles,  and  Turoff  contend,  “attention  to  instructional  design  is  one  of  the 
most  critical  factors  in  successful  learning  networks...  All  education  involves 
intervention  by  an  expert  to  organize  the  content,  sequence  instructional 
activities,  structure  task  and  group  interaction,  and  evaluate  the  process  (p.  20).” 

Bloom  (1956)  describes  curriculum  development,  or  instructional  design, 
as  a  four-step  process  that  begins  with  asking  what  educational  purposes  or 
objectives  the  course  seeks  to  attain?  Once  answered,  the  developer  must 
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create  learning  experiences  which  will  likely  bring  about  the  attainment  of  the 
objectives.  Next,  the  course  should  be  sequenced  to  enhance  transfer  and  avoid 
isolated  learning  experiences.  Finally,  the  effectiveness  of  the  learning 
experiences  must  be  evaluated  (p.  18).  In  their  faculty  guide,  Boetcher  and 
Conrad  (1999)  develop  a  comprehensive  four-part  model  for  developing 
instruction  that  does  not  conflict  with  Bloom’s  early  model.  The  model’s 
components  are  comprised  of  Analysis,  Learning  Objectives,  Selection,  and 
Evaluation  (Boetcher  and  Conrad,  1999,  p.  1 7). 

1.  Analysis 

The  first  step  in  the  instructional  design  process  is  to  conduct  an  analysis 
of  the  students  and  what  they  will  be  learning.  A  teacher  must  know  who  the 
students  are,  what  they  already  know  or  do  not  know,  and  what  they  will  need  to 
know  to  consider  the  training  program  a  success. 

In  his  book  Educative  Assessment,  Wiggens  (1998)  defines  a  standard  as 
something  that  describes  a  specific  and  desirable  level  or  degree  of  exemplary 
performance  irrespective  of  whether  most  people  can  or  cannot  meet  it  (p.106). 
He  also  contends  that  for  any  education  or  training  program  to  achieve  success, 
three  standards  must  exist.  First,  Content  Standards  need  to  be  in  place.  This 
describes  what  the  student  should  know  and  be  able  to  do  (p.  106).  Second, 
Performance  Standards  need  to  be  created.  This  standard  describes  how  well 
students  must  do  their  work.  Third,  Task  Standards  must  be  implemented.  This 
describes  what  tasks  the  students  must  be  able  to  perform  (p.  106). 

It  is  important  that  designers  realize  that  without  analyzing  what  the 
training  program  is  intended  to  accomplish,  what  the  students  need  to  know,  and 
how  to  best  teach  the  students  what  you  want  them  to  learn,  the  training  initiative 
may  very  well  fail. 

2.  Learning  Objectives 

The  learning  objectives  become  the  standard  by  which  the  training 
program  is  judged.  Poorly  defined  learning  objectives  hinder  the  evaluation 
process,  of  both  the  students  and  the  program.  Learning  objectives  formally 
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define  what  it  is  the  students  are  expected  to  learn  as  a  result  of  participating  in 
the  training  program. 

3.  Selection 

Selection  describes  the  process  of  choosing  teaching  strategies,  picking 
the  content  that  will  be  taught,  and  selecting  the  methods  that  will  be  used  to 
assess  the  program.  More  than  one  teaching  strategy  may  be  appropriate  for 
teaching  a  particular  class.  For  example,  an  instructor  may  decide  to  use 
multiple  teaching  strategies  by  using  Internet  delivery  methods  coupled  with 
several  classroom  lectures.  Content  selection  is  very  important.  The  instructor 
should  verify  that  the  course  content  would  satisfy  the  learning  objectives. 
Methods  used  to  assess  both  the  course  and  the  students  should  be  selected. 
This  means  the  developer  must  decide  what  is  to  be  evaluated,  how  it  is  to  be 
evaluated,  and  when  it  will  be  evaluated.  Once  this  is  decided,  the  developer 
must  decide  what  to  do  with  the  information  collected  via  the  evaluations. 

4.  Evaluation 

There  are  two  fundamental  evaluation  types:  Those  performed  by  the 
instructor,  and  those  performed  by  the  institution.  Instructors  generally  evaluate 
student  performance  through  observation,  quizzes,  and  examinations. 
Institutions  usually  assess  the  curriculum,  the  instructor,  or  the  program.  Opinion 
surveys,  questionnaires,  and  direct  observation  by  organizational  representatives 
are  common  methods  of  evaluation  employed  by  institutions  when  evaluating 
program  effectiveness. 

B.  PEDAGOGICAL  DESIGN 

All  educational  curriculums  must  be  properly  delivered  if  their  content  is  to 
be  understood,  remembered,  and  used  later  by  the  students.  Even  the  best 
instructional  designs  can  be  rendered  impotent  if  the  content  is  not  efficiently  and 
effectively  presented  to  the  students.  It  is  not  difficult  to  envision  a  teacher 
presenting  otherwise  interesting  material  to  students  in  a  way  that  is  so  boring 
most  students  in  the  class  are  uniformly  wondering  when  the  lesson  will  be 
mercifully  over  (how  good  the  instructional  design  was  becomes  irrelevant  if  and 
when  this  ever  occurs).  A  poor  teacher  can  easily  render  an  entire  course 
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ineffectual  through  poorly  designed,  or  if  designed  well,  poorly  executed  delivery 
methods.  To  be  successful,  different  delivery  methods  must  employ  different 
tactics  to  gain  the  students’  attention,  peak  their  interest,  and  motivate  them  to 
learn  the  material.  The  traditional  classroom  environment  emphasizes  different 
techniques  than  does  the  computer-based  environment,  although  the  ultimate 
goal  is  the  same. 

1.  Classroom  Instruction  versus  Computer-based  Delivery 

Traditional  instruction  and  computer-based  training  both  have  advantages 
and  disadvantages.  A  course  developer  must  be  familiar  with  the  pros  and  cons 
of  each  technique  in  order  to  select  a  delivery  method  that  best  suites  the 
organization’s  needs.  The  traditional  classroom  environment  features 
information  taught  by  an  instructor  or  through  a  textbook.  This  information  is 
taught  linearly,  or  in  a  set  sequence.  Classroom  training  is  synchronous, 
meaning  that  communications  take  place  in  real-time.  Web-based  and  computer 
based  training  can  provide  information  from  varied  sources,  many  times  via  a 
multimedia  presentation  that  makes  use  of  hyperlinks.  A  hyperlink  is  an 
embedded  doorway  that  when  clicked  on  with  the  mouse  pointer,  takes  the 
student  to  the  webpage  associated  with  that  link.  Hypermedia  is  the  use  of 
multiple  hyperlinks,  and  through  the  use  of  those  links,  allow  the  students  to 
control  where  they  go  within  the  framework  of  the  lesson  in  order  to  obtain  their 
learning  material.  Multimedia  is  the  use  of  two  or  more  forms  of  communication 
(e.g.  pictures,  sounds,  video,  etc)  to  present  information  to  a  target  audience  via 
a  computer.  Communication  with  the  instructor  can  be  asynchronous  or 
synchronous.  Asynchronous  means  that  there  is  a  delay  between  the  time  a 
communication  is  sent  and  the  time  it  is  received.  Electronic  mail  (email)  is  an 
example  of  asynchronous  communication.  Table  1  highlights  the  differences 
between  traditional  training  and  web-based  training  methods. 
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Lecture-Textbook 
Learning  Paradigm 

Web-based  Learning 
Paradigm 

Main  Sources  of 
Information 

Teacher  or  Textbook 

Various  Internet  Resources 
including  online  textbooks 

Format  of  Information 

Mostly  Lecture  or  Text 

Mostly  Multimedia 

Presentation  Format 

Linear 

Mostly  Hypermedia 

Interaction  Type 

Synchronous 

Asynchronous/Synchronous 

Table  1 .  Differences  between  Lecture-textbook  and  web-based  learning  paradigms 

Horton  (2000)  describes  several  advantages  to  instructor-led  training: 

(a)  The  instructor  can  answer  questions  and  solve  problems  as  they  arise  (p.  55). 

(b)  Instructors  provide  authority  that  some  learners  may  need  for  motivation  (p. 
55). 

(c)  An  instructor  can  adjust  the  course  to  suit  the  needs  of  a  particular  class  (p. 
55). 

(d)  Instructors  can  grade  the  activities  and  tests  too  subtle  for  automated  scoring 
(P-55). 

In  my  opinion,  the  ability  of  instructors  to  motivate  the  students  to  learn  is 
the  most  important  of  these  characteristics.  Since  an  unmotivated  student  is 
likely  to  learn  less  than  a  motivated  student,  it  is  very  important  that  instructors 
are  able  to  effectively  perform  this  function.  This  does  not  mean  an  instructor 
needs  to  take  on  the  role  of  an  enforcer,  as  the  connotation  of  motivator  might 
imply.  It  does  mean  that  the  teacher  is  able  to  provide  a  rich,  interesting,  and 
safe  learning  environment  that  attracts  and  inspires  students  to  learn. 

Traditional  classroom  instruction  also  possesses  a  few  inherent  problems. 
First,  not  all  instructors  are  created  equal.  Multiple  instructors  may  teach  the 
same  course  of  instruction,  which  means  students  taking  the  same  class  may  not 
receive  the  same  quality  of  instruction.  Second,  this  is  generally  considered 
“passive”  learning,  meaning  the  teacher  stands  in  front  of  the  class  and  lectures 
while  the  students  take  notes.  This  form  of  learning  is  not  very  interactive,  which 
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does  not  allow  the  students  much  time  to  mentally  work  with  the  information  and 
transfer  it  from  STSS  to  long-term  via  short-term  memory.  Consequently,  This 
style  of  teaching  may  not  produce  as  much  positive  transfer  among  the  student 
population  as  other  more  interactive  methods. 

Computer-based  training  possesses  several  advantages  over  traditional 
instruction.  In  Designing  Web-Based  Training,  Horton  (2000)  lists  several: 

(a)  Learners  develop  self-reliance  that  is  needed  after  the  class  (p.  55). 

(b)  Learners  are  not  required  to  conform  to  the  instructor’s  schedule  (p.  55). 

(c)  All  learners  get  the  same  quality  of  learning  experience  (p.  55). 

Developers  today  can  create  stunning  graphics  and  visual  effects,  embed 
video  and  audio,  and  produce  highly  entertaining  and  interesting  presentations 
that  grab  and  hold  students’  attention.  Coupled  with  the  advantages  listed  above 
by  Horton,  powerful  learning  tools  can  be  created  that  allows  the  students  to,  in 
essence,  take  control  over  their  own  learning. 

Despite  its  potential  powerful  advantages,  CBT  does  have  several 
possible  disadvantages.  One  of  weaknesses  of  computer-based  training  is  that  it 
is  generally  more  expensive  to  deliver  than  traditional  teaching  methods.  In 
today’s  fiscally  conscious  environment,  it  is  not  surprising  that  most  courses  and 
CBT  web  sites  are  not  very  interactive,  and  fall  well  short  of  initial  expectations 
(Parikh  and  Sameer,  2002,  p.  28). 

Another  potential  drawback  of  CBT  is  that  there  is  substantially  less 
instructor  control  and  struggling  students  may  not  get  the  much-needed  help  they 
require  to  successfully  complete  the  training.  If  students  feel  isolated,  or  are  not 
motivated  to  learn,  they  may  not  take  responsibility  for  their  coursework. 
Subsequently,  this  may  lead  to  a  high  rate  of  attrition.  Unmotivated  trainees  are 
unlikely  to  seek  help  if  cumbersome  processes  are  involved. 

Parikh  and  Sameer  (2002)  state  there  is  no  feedback  loop  between 
instructors  and  students  unless  a  technically  complex  and  cumbersome  login 
function  is  added  to  the  courseware  (p.  29).  This  type  of  secure  instructor- 

28 


student  feedback  loop  can  add  considerable  costs  to  a  CBT  or  web-based 
training  initiative.  However,  not  including  this  functionality  with  the  CBT 
courseware  may  lead  to  feelings  of  isolation  and  the  subsequent  motivation 
issues  discussed  in  the  previous  paragraph. 

Finally,  there  is  the  potential  to  rely  too  heavily  on  technology  when 
designing  CBT.  In  the  end,  technology  cannot  replace  the  human  teacher  in 
education.  Yeung  (2002)  believes  that  although  technology  can  be  used  to 
leverage  instructors’  time,  it  cannot  replace  most  human  contact  without 
significant  quality  losses. 

2.  Stolovich’s  Five-Step  Model  for  Training 

It  is  essential  to  formally  structure  the  training  students  receive.  This  is 
true  regardless  of  the  delivery  method,  as  it  reasonably  assures  the  organization 
that  all  students  will  receive  approximately  the  same  training  irrespective  of  who 
the  instructor  is.  Stolovitch  and  Keeps  (2003)  present  a  comprehensive  five-step 
model  for  structuring  training.  Figure  3  shows  this  model.  This  model  includes 
the  following  steps: 

(1)  Rationale.  Explain  what  it  is  you  want  the  students  to  learn.  This  serves  to 
motivate  the  students  to  learn  the  important  information  since  it  has  a  high  value 
associated  with  it  (p.  68). 

(2)  Objective.  Stolovitch  and  Keeps  (2003)  suggest  that  by  telling  students  what 
they  will  be  able  to  do  at  the  end  of  the  lesson,  there  is  a  better  chance  they  will 
learn  it  (p.  70). 

(3)  Activities.  Learning  activities  guide  students  toward  the  fulfillment  of  the 
stated  learning  objectives  (p.  71).  These  activities  should  be  stimulating, 
contributing  to  students’  experiences,  imaginations,  and  judgments  (p.  71). 

(4)  Evaluation.  Student  evaluation  verifies  what  the  students  have  learned.  It 
determines  the  degree  to  which  the  learner  has  met  each  objective  for  the 
desired  level  of  performance  (p.  72).  The  most  common  evaluative  techniques 
are  tests  and  performance  appraisals  (p.  73). 
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(5)  Feedback.  Done  correctly,  feedback  lets  the  learners  know  what  they  did 
correctly  and  provides  constructive  help  when  they  have  done  something  wrong 
(p.  73).  There  are  two  forms  of  feedback:  corrective  and  confirming.  Corrective 
feedback  explains  how  students  can  obtain,  or  meet,  a  stated  learning  objective 
(p.  73).  Confirming  feedback  is  a  reward  for  attaining  a  learning  objective  (p.  73). 


Figure  3.  Five-step  Model  for  Training  (From  Stolovitch  and  Keeps,  2003,  p.  69) 

Although  Stolovitch  and  Keeps  (2003)  present  a  fairly  comprehensive 
model,  it  lacks  a  function  that  verifies  the  instructional  interface  is  appropriate  for 
the  training  being  conducted.  It  is  important  to  examine  instructional  interface  in 
CBT  because  the  computer  becomes  the  teacher.  Flow  a  student  interacts  with 
the  computer  during  CBT  many  have  an  impact  on  whether  the  student  is  able  to 
realize  the  maximum  educational  benefits  of  the  training.  The  evaluation 
framework  presented  at  the  end  of  this  chapter  will  assess  instructional  interface 
in  CBT.  Lohr  and  Eikleberry  (2001)  define  instructional  interface  as,  “the 
elements  in  any  product  or  system  that  support  the  tasks  of  the  learner  while  he 
or  she  is  learning  (p.  25).”  Lohr  and  Eikleberry  (2001)  continue  by  stating  this 
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interface  can  be  human-based,  computer-based,  paper-based,  or  a  combination 
thereof  (p.25).  This  interface  is  the  link  between  the  student  and  the  information 
or  tasks  being  learned,  and  appears  to  be  directly  linked  to  the  teaching  method 
and  is  an  integral  part  of  answering  some  of  the  fundamental  questions  students 
ask,  such  as,  “Am  I  being  graded?”  or  “What  am  I  supposed  to  do  now?”  (Lohr 
and  Eikleberry,  2001,  p.  25).  When  a  CBT  system  lacks  human  instructors  that 
would  otherwise  guide,  inspire  and  answer  questions,  a  well-designed  and 
properly  functioning  instructional  interface  is  critical.  Instructional  Interface  will  be 
examined  in  more  detail  in  a  later  section  of  this  thesis. 

3.  Computer-based  Pedagogical  Design  Principles 

Distance  Learning  can  be  an  efficient  way  for  organizations  to  conduct 
training.  Computer-based  instruction  using  active,  self-paced  learning  with 
individualized  feedback  can  reduce  total  learning  time  by  30  to  40  percent 
(Desypris,  2002,  p.  13).  In  fact,  Horton  reports  that  California  State  University  at 
Northridge,  Office  Depot,  Merrill  Lynch,  and  Toys  R  Us  have  all  generated 
significantly  better  training  results  since  beginning  Distance  Learning  initiatives 

(p.  26). 

a.  Potential  Drawbacks 

Despite  these  successes  enjoyed  by  many  organizations,  CBT  is 
not  without  potential  problems.  It  is  possible  that  meanings  will  be  misinterpreted 
(Horton,  2000,  p.37).  If  a  student  does  not  adequately  understand  the  instruction 
then  it  is  likely  that  some  of  the  presentation  will  be  misinterpreted.  The  fact  that 
there  is  no  human  instructor  only  exacerbates  this  problem  if  it  occurs.  Another 
potential  problem  that  arises  from  the  use  of  CBT  is  that  there  are  very  few  ways 
to  get  a  student  “unstuck”  (Horton,  2000,  p.  37).  Horton  (2000)  also  raises  the 
fact  that  there  is  very  little  socialization  that  occurs  when  one  learns  alone  (p. 
37).  He  argues  that  this  leads  to  two  potential  problems.  First,  a  student  is 
unable  to  learn  from  his  or  her  classmates  (p.  37).  Second,  if  lonely  because  of 
insufficient  socialization,  a  student’s  motivation  will  decline  (p.  37).  Lastly,  there 
are  too  many  possible  disruptions  (p.  37).  As  one  who  has  first-hand  experience 
with  classes  that  have  had  much  of  the  required  information  presented  on-line,  I 
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can  attest  to  the  fact  it  was  often  difficult  to  find,  download,  read,  and  absorb  the 
material  in  a  timely  manner  due  to  all  the  interruptions  and  disruptions.  In  my 
opinion,  this  is  potentially  the  biggest  problem  area  of  all.  Unfortunately,  it  is  also 
the  one  sphere  of  influence  that  a  course  developer  has  the  least  amount  of 
control  over. 

b.  General  Issues 

Most  of  CBT’s  potential  pitfalls  can  be  avoided  by  taking  advantage 
of  the  development  principles  others  have  found  successful.  As  a  general  rule, 
Allessi  and  Trollip  (1991)  contend  that  the  course  developers  must  answer 
several  questions  during  the  development  process: 

(a)  Is  the  use  of  the  computer  appropriate  (p.486)? 

(b)  Is  the  methodology  appropriate  (p.  486)? 

(c)  Are  the  lesson  lengths  appropriate  (p.  486)? 

(d)  Is  the  mastery  level  appropriate  for  each  topic  (p.  486)? 

c.  Student  Interaction 

Once  these  questions  are  satisfactorily  answered,  developers  can 
critically  examine  other  aspects  of  their  initial  design.  How  the  students  will 
interact  with  the  computer  is  an  issue  developers  must  closely  scrutinize. 
Belanger  and  Jordon  (2000)  contend  that  interactivity  is  the  key  element  of  the 
learning  environment,  and  define  it  as  the  degree  of  intellectual,  emotional,  and 
physical  engagement  of  the  learner  to  the  instructional  content  using  computer- 
based  Distance  Learning  technologies  (p.  23).  Belanger  and  Jordon  (2000) 
describe  four  successively  more  complex  levels  of  student  interactivity.  They  are 
from  least  to  most  complex:  Passive,  Limited  Participation,  Complex 
Participation,  and  Real-time  participation  (p.  134).  Passively  oriented  text  and 
graphics  are  not  generally  considered  the  optimal  choice  for  use  when 
developing  an  integrated  CBT  program.  This  method  can  be  used  for  teaching 
sets  of  basic  facts  or  rules  (Belanger  and  Jordon,  2000,  p.  1 34). 

Interactive  text  presents  learners  with  different  pages  of  content, 
depending  on  the  selections  they  make  during  the  course  of  the  lessons. 

32 


Belanger  and  Jordon  (2000)  recommend  using  an  index  or  hyperlinks  for 
navigation  (p.  138).  By  using  hyperlinks  throughout  a  lesson,  information  from 
one  lesson  can  be  linked  to  information  in  other  lessons.  Belanger  and  Jordon 
(2000)  believe  interactive  text  and  graphics  reinforces  the  ways  in  which  the 
material  relates  to  other  topics.  The  amount  of  student  transfer  increases 
because  facts  and  processes  just  learned  are  immediately  practiced  (p.  1 39). 

The  most  expensive,  complex,  and  fully  interactive  type  of  distance 
learning  instruction  is  interactive  multimedia  (Belanger  and  Jordon,  2000,  139). 
Belanger  and  Jordon  (2000)  explain  that  when  using  interactive  multimedia, 
interactive  text  is  combined  with  video,  sound,  animation,  and  high-resolution 
graphics  to  provide  a  rich  learning  environment  (p.  139).  According  to  Belanger 
and  Jordon  (2000),  a  highly  interactive  CBT  environment  decreases  students’ 
sense  of  isolation,  allows  more  senses  to  be  used  during  the  learning  process, 
and  students  are  engaged  vice  learning  through  passive  listening,  which 
increases  interest  and  motivation  (p.  21). 

It  is  important  that  developers  properly  select  the  most  appropriate 
level  of  interactivity  with  respect  to  organizational  training  objectives.  Different 
lessons,  or  even  different  parts  of  the  same  lesson,  may  require  different  levels 
of  interactivity.  These  requirements,  linked  back  to  the  learning  objectives,  are 
what  drive  the  interactive  complexity  of  each  topic. 

Allessi  and  Trollip  (1991)  recommend  that  developers  ask  five  basic 
questions  about  how  students  will  interact  with  the  computers  when  deciding  on 
how  interactive  to  make  a  lesson: 

(a)  Is  interaction  frequent  (p.486)? 

(b)  Is  comprehension  enhanced  (p.  486)? 

(c)  Is  memory  enhanced  (p.  486)? 

(d)  Is  transfer  enhanced  (p.  486)? 

(e)  Are  there  a  variety  of  student  interactions  with  the  computer  (p.  486)? 
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If  these  questions  cannot  be  answered  in  the  affirmative,  then  it  is 
probable  that  students  will  be  victimized  by  at  least  one  of  the  problems  listed  in 
this  chapter,  such  as  boredom,  low  motivation,  misinterpretation  of  instructional 
material,  inability  to  proceed  any  further,  etc.  Frequent  interaction  with 
interesting  material  keeps  students’  interests  high  while  simultaneously  providing 
motivation  to  learn.  From  experience,  a  series  of  long  PowerPoint  presentations, 
where  the  students  only  interaction  is  hitting  the  enter  key  to  advance  along  one 
slide  at  a  time  is  neither  interesting  nor  motivating. 

Another  approach  used  to  assess  this  aspect  of  design  is  to  ask  if 
there  is  a  teacher  in  the  interface  (Lohr  and  Eikleberry,  2001,  p.  24).  This  takes 
into  account  the  fact  a  human  instructor  is  absent  while  the  student  is  actively 
learning.  By  frequently  asking  this  question  while  designing  a  course,  developers 
can  build-in  software  applications  that  take  on  the  role  of  an  instructor,  or  an 
assistant,  while  students  are  using  the  program. 

d.  Human-Computer  Interface 

There  are  many  different  types  of  human-computer  interaction 
styles  which  developers  can  implement  for  use  in  a  training  program.  It  is  best  if 
one  style  is  selected  and  used  throughout  the  entire  range  of  associated  training 
topics  instead  of  changing  styles  from  topic  to  topic.  Changing  styles  will  only 
confuse  and  frustrate  the  students  taking  the  class.  In  fact,  Horton  (2000)  argues 
that  designers  should  modularize  the  course,  separating  it  by  loosely  associated 
topics.  This  will  minimize  the  dependence  between  topics  (p.  174).  Whichever 
method  is  used,  the  students  will  need  to  use  an  input  device  (e.g.  mouse, 
keypad,  etc)  to  interact  with  the  computer.  Allessi  and  Trollip  (1991)  contend  that 
designers  must  carefully  select  the  method  of  inputting  selections  into  the 
computer  and  that  the  selected  method’s  interface  should  assist  in  preventing  or 
detecting  any  student  errors.  Some  of  the  more  popular  styles  include  Menus, 
WIMP  (Windows,  Icons,  Menus,  and  Pointers),  and  Point  and  Click  (Dix  et  al., 
1997,  p.  115). 

Dix  et  al.  (1997)  describe  menus  as  a  set  of  on-screen  options  that 

are  selected  using  an  input  device.  For  menus  to  be  beneficial  to  the  students, 
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menu  options  must  be  meaningful  and  logically  grouped  together  (Dix  et  al., 
1997,  117).  Horton  (2000)  lists  three  suggestions  for  making  a  menu  driven 
interface  more  effective.  First,  build  in  an  index  and  search  function  so  learners 
have  more  ways  to  find  related  topics  (p.  174).  Second,  display  menus  in 
separate  frames.  Learners  can  then  navigate  to  related  topics  via  the  menu 
rather  than  by  links  within  the  main  body  of  the  web  or  course  pages  (p.  174). 
Third,  automate  sequential  navigation.  Develop  the  software  so  the  “next”  and 
the  “previous”  buttons  calculate  where  to  go  based  on  an  easily  maintained  list  of 
topics  (p.  174). 

Dix  et  al.  (1997)  portray  the  WIMP  interface  as  an  interface  that 
contains  windows,  icons,  menus,  and  pointers.  It  is  similar  to  the  Windows 
based  operating  system  that  is  found  on  most  of  today’s  personal  computers. 

Point  and  click  styles  utilize  a  menu  that  students  use  a  pointer 
(mouse)  and  click  on  the  option  that  they  desire  to  initialize  (Dix  et  al.,  1997, 
p.129).  Horton’s  suggestions  regarding  the  use  of  menus,  such  as  building  an 
index  and  search  function,  using  separate  frames,  and  automating  navigation  are 
also  appropriate  to  this  interface  style. 

e.  Program  Control  Issues 

There  is  a  wide  array  of  non-instructional  control  issues  that  must 
be  resolved  prior  to  students  participating  in  a  computer  delivered  course  of 
instruction.  These  issues  include  directions,  book  marking  during  a  lesson, 
finishing  a  lesson,  and  moving  forward  and  backward  throughout  a  lesson. 
Methods  available  for  students  to  use  when  activating  one  of  these  control 
functions  can  include  menus,  icons,  or  special  keys  on  the  keyboard  that  initiate 
specific  functions.  Allessi  and  Trollip  (1991)  recommend  using  menus  for  such 
things  as  initial  control,  sequence  selection,  and  review  functions  (p.  26).  Allessi 
and  Trollip  (1991)  further  advocate  saving  command  key  usage  for  forward 
progression,  backward  paging,  help,  index,  and  exit  functions  (p.  26).  While 
Belanger  and  Jordan  (2000)  do  not  discriminate  between  menus  and  command 
keys,  they  argue  that  learners  must  be  provided  with  the  ability  to  add,  delete, 
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and  organize  bookmarks;  track  progress;  use  a  review  function;  and  the  ability  to 
generate  and  use  practice  quizzes  and  tests  (p.  149). 


Every  student  needs  the  capability  to  find  and  read  the  courseware 
directions.  Allessi  and  Trollip  (1991)  claim  directions  are  essential  to  the 
effectiveness  of  any  computer-based  lesson  (p.  21).  Whether  directions  are 
optional  or  mandatory,  Allessi  and  Trollip  argue  that  it  is  vital  to  ensure  they  are 
readily  available  (p.  21).  This  makes  sense.  If  optional  directions  were  not  easy 
to  find  and  use,  students  would  quickly  give  up  and  the  lesson  would  not  be 
satisfactorily  completed.  A  paper-based  manual  may  be  the  best  method  for 
delivering  directions  to  students.  This  manual  can  be  saved  on  a  server  in  an 
Adobe  PDF  format,  and  then  downloaded  via  the  Internet  by  students.  Students 
will  be  able  to  more  easily  access  and  use  directions  in  the  event  of  software 
“glitches”  if  the  directions  are  obtained  separately  and  used  independently  from 
the  courseware. 

From  a  student’s  perspective,  the  ability  to  bookmark  a  spot  in  a 
lesson,  leave,  and  then  come  back  to  it  is  important.  Students  may  not  have  the 
time  to  successfully  complete  a  given  lesson  in  one  sitting.  No  student  wants  to 
start  a  lesson,  have  to  leave  in  the  middle  of  that  lesson,  and  then  have  to  restart 
from  the  beginning  again  when  they  came  back.  This  is  a  waste  of  time  and 
effort.  Consequently,  the  ability  to  bookmark  a  lesson  is  critical  to  students’ 
attitudes  toward  their  learning  environment,  and  quite  possibly  to  their  successful 
completion  of  a  course  of  i  nstruction. 

Allessi  and  Trollip  (1991)  recommend  students  be  given  the  option 
to  quit  the  program,  return  to  the  main  menu,  or  continue  to  the  next  planned 
lesson  whenever  they  complete  a  topic  (p.  85).  Empowering  the  students  to 
decide  what  topic  to  complete  next  gives  the  students  a  sense  of  control,  and 
improves  motivation  to  continue  using  the  software. 

Another  vital  control  function  students  need  is  the  ability  to  page 
forward  and  backward  throughout  a  lesson.  Sometimes  a  student  is  very  familiar 
with,  or  has  already  mastered,  the  particulars  of  part  of  a  topic.  The  student  may 
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want  to  rapidly  scan  through  the  portion  of  material  he  or  she  has  mastery  of,  and 
the  incorporation  of  a  page  forward  key  or  icon  is  a  good  way  to  give  this  control 
to  the  student.  Alternately,  if  a  student  has  not  understood  the  material  over  the 
course  of  several  slides  in  a  topic,  it  is  essential  to  give  that  student  the  ability  to 
go  back  and  reexamine  those  slides.  In  this  way,  a  student  can  repeatedly  go 
through  any  section  of  a  lesson  until  it  is  understood  or  mastered. 

f.  Design  Evaluation 

Once  an  organization  has  developed  an  initial  design  for  a  training 
program,  it  is  best  to  evaluate  that  design.  Dix  et  al.  (1997)  offer  many  useful 
ideas  about  evaluating  designs  that  can  be  useful  in  determining  if  the  initial  CBT 
blueprint  is  correct.  He  suggests  the  CBT  system  should  always  keep  users 
informed  about  what  is  happening  via  timely  feedback.  Since  consistency  is  key, 
developers  should  verify  that  users  do  not  have  to  wonder  whether  different 
words,  situations,  or  actions  mean  the  same  thing.  The  user  should  not  have  to 
remember  information  from  one  part  of  a  dialog  to  another  and  instructions 
should  always  be  available.  Despite  the  prevalence  of  computers,  there  are  still 
many  people  who  have  a  difficult  time  operating  them  (p.  414).  According  to  Dix 
et  al.  (1997),  it  is  also  important  that  all  CBT  systems  be  able  to  help  users 
recognize,  diagnose,  and  recover  from  errors  (p.  414).  By  verifying  a  CBT 
program  is  designed  to  satisfactorily  meet  these  criteria,  developers  can  help 
ensure  a  relatively  smooth  transition  from  the  design  phase  of  a  CBT  initiative  to 
all  subsequent  phases  with  minimal  design  rework. 

4.  Lesson  Organization  and  Sequencing 

If  a  training  initiative  is  going  to  be  effective,  it  must  be  efficiently 
organized  and  the  lessons  should  be  properly  sequenced.  In  other  words,  the 
effort  must  be  properly  framed.  Horton  (2000)  describes  a  course’s  framework 
as  the  routine  parts  of  the  course  that  do  not  teach  subject  matter  but  are 
nevertheless  an  essential  part  of  the  course  (page  78).  Allessi  and  Trollip 
(1991),  along  with  Horton  (2000),  agree  that  these  can  include,  but  are  not 
limited  to  the  following:  Course  announcements  page,  course  and  lesson 
introduction  pages,  learning  objectives  pages,  learning  materials,  and  a  feedback 
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loop.  Some  authors,  such  as  Allessi  and  Trollip  (1991),  include  the  course’s 
actual  instructional  content  within  a  course’s  framework. 

a.  Announcements  Page 

Although  it  may  appear  unnecessary  to  develop  an  announcements 
page  if  conducting  training  exclusively  with  CBT,  it  is  still  a  good  idea  to 
implement.  There  are  two  basic  reasons  why.  First,  just  because  the  training  is 
conducted  via  the  computer  does  not  necessarily  mean  that  testing  is.  An 
announcements  page  is  a  convenient  method  to  announce  quizzes,  tests,  and 
performance  appraisals.  Second,  the  instructional  information  on  a  particular 
compact  disc  may  need  to  be  changed.  An  announcements  page  on  the  Internet 
allows  administrators  to  easily  pass  all  relevant  information  regarding  changes  to 
the  students. 

b.  Introduction 

In  CBT,  the  introductory  part  of  a  lesson  consists  of  two  parts.  The 
first  is  the  title  page.  The  title  page  is  used  to  attract  attention,  create  a  receptive 
attitude,  and  to  indicate  what  the  topic  is  about  (Allessi  and  Trollip,  1991,  p.  19). 
An  unattractive,  sloppy,  or  overly  simplistic  title  page  may  give  students  an  idea 
the  presentation  is  going  to  be  unattractive,  sloppy,  or  boring,  which  will  decrease 
students  motivation  to  progress  through  the  lesson  before  any  learning  has 
occurred. 

The  second  part  of  the  introduction  is  the  presentation  of  learning 
objectives.  As  indicated  previously,  learning  objectives  indicate  what  the  student 
will  be  able  to  do,  say,  or  write  at  the  completion  of  the  lesson  (Allessi  and  Trollip, 
1991,  p.  19).  For  example,  a  typical  learning  objective  may  look  similar  to,  “After 
this  lesson,  you  will  be  able  to  set  the  time  on  your  VCR.”  Allessi  and  Trollip 
(1991)  identify  two  types  of  learning  objectives,  behavioral  and  non-behavioral. 
A  behavioral  objective  expresses  the  task,  conditions,  and  performance  required 
by  the  student.  Reporting  contact  with  another  ship  to  the  Captain  is  an  example 
of  a  task  routinely  performed  at  sea.  The  behavior  objective  for  this  task  would 
include  the  prerequisite  conditions  that  must  exist  prior  to  starting  the  procedure, 
data  to  be  gathered  during  the  procedure,  how  to  arrange  the  data  so  it  becomes 
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useful  information  to  the  Captain,  and  what  to  say  when  actually  making  the 
report.  Non-behavioral  objectives  enhance  focus  on  both  specific  and  non¬ 
specific  aspects  of  a  lesson.  I  believe  using  behavioral  objectives  in  a  training 
environment  is  the  better  organizational  policy.  In  doing  so,  the  students  remain 
focused  on  the  most  important  aspects  of  their  training.  Horton  (2000)  advocates 
writing  the  objectives  in  terms  of  what  the  student  will  do,  say,  or  write  as 
opposed  to  providing  the  learning  objectives  as  teaching  cbjectives  (from  an 
instructors  point  of  view),  which  stresses  the  application  of  training  in  a 
meaningful  way  to  the  students  (p.  91).  Horton  (2000)  identifies  phrases  and 
words  that  are  good  to  use,  and  those  that  are  undesirable  when  writing  learning 
objectives.  When  generating  learning  objectives,  Horton  (2000)  recommends 
using  words  such  as  create,  make,  install,  set-up,  edit,  revise,  amend,  start, 
repair,  replace,  diagnose,  troubleshoot,  organize,  and  write  (p  91).  Words  that 
should  be  avoided  include  understand,  list,  give  an  example  of,  articulate, 
explain,  describe,  identify,  recall,  master,  and  differentiate  (p.  91).  According  to 
Horton  (2000),  these  words  are  to  be  avoided  because  they  describe  student 
performance  in  terms  of  what  the  instructor  is  teaching,  not  in  terms  of  what  the 
student  must  do  to  meet  the  objective. 

c.  Learning  Materials  /Instructional  Information 

Learning  materials  include  books,  manuals,  videotapes,  DVD’s, 
CD-ROM’s,  and  all  other  media  that  assists  the  student  in  learning.  Each  of  the 
aforementioned  media  all  have  one  thing  in  common.  They  each  present 
instructional  information  to  the  student.  How  they  do  this  is  also  important. 

According  to  Allessi  and  Trollip  (1991),  there  are  four  types  of 
information  that  can  be  effectively  presented  to  students  via  CBT.  They  are  (a) 
verbal  information,  (b)  general  concepts,  (c)  rules  and  principles,  and  (d)  skills  (p. 
44).  I  have  reservations  about  the  utility  of  teaching  skills  by  way  of  CBT.  Of 
these  four  types,  a  set  of  skills  is  the  most  difficult  to  teach  via  the  CBT  method. 
In  order  to  effectively  teach  a  new  skill  set,  it  may  be  necessary  to  teach  the 
concepts,  rules,  and  principles  that  guided  the  development  of  the  skill.  CBT  is 
an  appropriate  and  proven  vehicle  to  present  the  first  three  types  of  information, 
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but  it  may  not  be  the  best  mechanism  to  teach  skills,  especially  if  CBT  is  used  in 
conjunction  with  on-the-job-training.  From  experience,  it  is  easier  to  learn  a  new 
procedure,  especially  when  it  involves  operating  machinery,  when  the  trainee  is 
able  to  see,  touch,  hear,  smell,  and  manipulate  the  actual  equipment.  As  a 
general  rule,  it  is  best  to  teach  from  the  simple  to  the  complex.  This  means 
ideas,  concepts,  then  rules  and  principles,  followed  by  the  actual  procedure 
should  be  the  sequence  in  which  skills  are  taught. 

d.  Feedback  Loop 

The  feedback  loop  is  a  very  powerful  learning  tool  educators  have 
at  their  disposal.  According  to  Stolovitch  and  Keeps  (2003),  Allessi  and  Trollip 
(2001),  and  Wiggins  (1998),  feedback  to  students  should  depend  on  their 
answers  and  needs  to  be  either  corrective  or  confirming.  Wiggins  (1998)  states 
corrective  feedback  will  compare  current  student  performance  to  what  a 
successful  outcome  is,  which  focuses  student  attention  on  how  to  improve  (p. 
49).  Stolovitch  and  Keeps  (2003)  argue  confirming  feedback  is  used  as  a  reward 
for  answering  questions  correctly,  and  that  frequent  and  specific  feedback  has 
been  demonstrated  to  improve  performance  (p.  67). 

5.  Instructional  Presentation 

There  are  three  basic  aspects  to  a  CBT  delivered  presentation  I  will 
examine  in  this  section.  They  are  the  screen  organization  and  use  of  graphics; 
text,  language,  and  grammar;  and  mid-topic  questioning  techniques.  Screen 
organization  centers  around  the  layout  of  the  screen  during  a  lesson. 
Additionally,  basic  guidelines  for  the  use  of  graphics  will  be  presented.  The 
proper  use  of  language  and  grammar  are  essential  to  the  development  of  a  high- 
quality  training  program.  The  way  in  which  textual  information  is  included  in  a 
presentation  can  affect  learning.  Simple  guidelines  for  presenting  text  will  be 
explained.  An  important  aspect  to  any  training  program  is  the  use  of  questions 
throughout  the  lesson  to  determine  student  comprehension  and  the  associated 
feedback  to  the  students  about  their  performance.  Successful  questioning 
techniques  will  also  be  examined. 
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a.  Screen  Organization 

According  to  Berry  (2000),  research  suggests  that  visuals  (web 
pages  or  CBT  pages)  are  equivalent  to  lengthy  text  in  their  ability  to  effectively 
communicate  (p.  48).  Leflore  (2000)  details  three  laws  that  are  important  to  know 
if  developing  CBT  software.  By  understanding  the  principles  encompassed  by 
these  laws,  developers  can  begin  to  place  instructional  objects  into  courseware 
in  ways  that  are  very  beneficial  to  the  students  they  are  training.  The  first  is  the 
Law  of  Proximity,  which  states  it  is  easier  for  learners  to  understand  that  different 
text  or  graphic  elements  go  together  if  these  elements  are  placed  close  together 
(p.  103).  For  example,  the  text  that  describes  a  figure  or  illustration  should  be 
placed  close  to  that  illustration.  The  second  law  Leflore  (2000)  describes  is  the 
Law  of  Similarity  (p.  103).  This  law  states  that  people  will  group  things  together 
that  look  similar  (p.  103).  For  example,  if  elements  in  a  graphic  are  all  of  the 
same  style,  the  graphic  will  be  seen,  or  perceived,  as  a  whole  (p.  103).  The  third 
law  Leflore  (2000)  mentions  is  the  law  of  Closure.  Leflore  (2000)  states 
individuals  will  try  to  interpret  incomplete  graphics  or  text  based  upon  past 
experience  (p.  104). 

Leflore  (2000)  also  provides  a  set  of  six  Gestalt  Theory  guidelines 
appropriate  for  following  when  constructing  a  CBT  delivered  course.  They  are: 

(a)  When  designing  web-based  instruction,  make  sure  that  the  background  does 
not  interfere  with  the  clarity  of  the  information  presented  in  the  foreground  (p. 
104). 

(b)  Use  simplified  graphics  to  introduce  new  material.  If  the  concept  requires 
complexity,  start  simple  and  gradually  add  in  the  complexity  (p.  104). 

(c)  Place  related  information  together  on  the  screen  so  the  learner  will 
automatically  grasp  the  information  together  as  a  unit  rather  than  as  separate 
elements  (p.  104). 

(d)  Use  color,  animation,  flashing,  or  other  means  to  draw  attention  to  key 
phrases  in  text  or  areas  in  graphics  (p.  1 04). 
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(e)  Ensure  text  and  graphics  are  complete,  so  the  student  does  not  have  to 
spend  energy  “making”  or  “creating”  meaning  (p.  1 04). 

(f)  When  introducing  a  new  topic,  vocabulary  unique  to  the  content  should  be 
introduced  by  using  common  terms.  Avoid  using  jargon  until  after  all  the  new 
terms  have  been  introduced  (p.  104).  As  a  matter  of  style,  it  is  important  that 
courseware  developers  avoid  using  acronyms  without  first  explaining  them 
(Allessi  and  Trollip,  1 991 ,  p.  43). 

It  is  disconcerting  to  students  when  they  must  “scroll”  through 
lengthy  text.  According  to  Allessi  and  Trollip  (1991),  it  is  hard  for  people  to 
distinguish  between  old  and  new  information  when  it  is  moving  (p.  35). 
Therefore,  it  is  important  to  limit  the  amount  of  textual  information  that  is 
presented  to  no  more  than  one  page  at  a  time  to  avoid  making  the  students  scroll 
through  text.  Allessi  and  Trollip  (1991)  also  recommend  enclosing  the  primary 
text  in  a  box,  for  emphasis,  when  using  both  graphics  and  text  on  the  same  page. 
The  students’  attention  will  then  become  focused  on  the  most  important 
information  (p.  38). 

b.  Graphics 

Graphics  are  very  useful  tools  that  can  be  used  for  the  presentation 
of  instructional  information,  presentation  of  analogies,  or  as  cue’s  for  student 
action.  Belanger  and  Jordon  (2000)  detail  two  types  of  graphical  animation: 
bitmapped  or  object-oriented.  A  bitmapped  image  is  drawn,  and  motion  is 
generated  through  showing  progressive  movement  in  frames  per  second  (FPS). 
Full  motion  video  is  shown  at  30  FPS  and  a  minimum  of  10  FPS  is  required  to 
avoid  a  “choppy”,  or  strobe  effect  (Belanger  and  Jordon,  2000,  p.  1 30). 

Complex  graphics  can  easily  confuse  students  unfamiliar  with  the 
material.  Graphics  that  contain  a  lot  of  information  should  be,  if  possible,  broken 
down  into  smaller  parts  and  presented  sequentially  (Allessi  and  Trollip,  1991,  p. 
40).  Allessi  and  Trollip  (1991)  also  contend  that  graphics  should  always  be 
presented  with  their  associated  text  because  changing  back  and  forth  between 
an  illustration  and  its  textual  description  makes  comprehension  more  difficult  for 
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the  student  (p.  40).  According  to  Williams,  Paprock,  and  Covington  (1999), 
designers  can  also  help  the  students  by  using  graphical  displays  that  are  “off- 
centered”.  This  forces  the  students  to  scan  the  whole  screen  and  not  just  focus 
on  the  center  of  it  (141). 

1.  Color.  The  use  of  color  can  add  breadth  to  any 
presentation,  increasing  student  interest  and  motivation.  For  example,  using 
different  colors  to  represent  segments  of  a  pie  chart  can  be  an  effective  way  to 
help  students  quickly  distinguish  between  each  segment  of  the  graph.  Using  a 
colored  screen  may  indicate  to  students  that  an  action  must  be  taken.  If  the 
computer  is  awaiting  student  input,  the  background  color  of  the  screen  could  be 
yellow.  The  screen  could  flash  red  or  green  depending  on  whether  a  student 
answered  a  question  correctly.  Although  the  use  of  color  is  encouraged,  there 
are  a  few  rules  to  be  followed  when  using  color  in  a  CBT  environment. 

Allessi  and  Trollip  (1991),  and  Williams,  Paprock,  and 
Covington  (1999)  agree  that  if  multiple  colors  are  used  on  the  same  screen,  they 
should  contrast.  Additionally,  red  and  blue  should  be  used  sparingly  since  they 
are  not  perceived  as  well  as  colors  in  the  middle  of  the  visible  spectrum,  such  as 
yellow  and  green.  Dix  et  al.  (1997)  add  that  the  colors  used  should  correspond 
to  common  conventions  and  user  expectations  (p.  115).  This  means  red  should 
mean  “wrong”,  or  “stop”,  whereas  green  should  mean  “correct”,  or  “go”.  Dix  et  al. 
(1997)  also  argue  if  color  is  used  as  an  indicator,  it  should  not  be  the  only  cue 
given  to  the  student.  For  example,  if  a  course  used  a  colored  screen  to  cue  a 
student  that  a  question  is  answered  correctly  or  not,  it  should  also  include  the 
words  “Correct”,  or  “Wrong”  within  the  colored  screen  as  a  second  cue  to  the 
student. 

c.  Text,  Language,  and  Grammar 

There  are  a  number  of  factors  that  must  be  considered  when 
presenting  textual  information  via  a  computer.  These  include  the  characteristics 
of  the  text  itself,  textual  quality,  along  with  language  and  grammar. 

Williams,  Paprock,  and  Covington  (1999)  offer  a  set  of  guidelines 

appropriate  for  the  way  text  should  “look”,  or  its  characteristics,  when  using  CBT 
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to  present  information.  Williams,  Paprock,  and  Covington  (1999)  state  that  since 
the  art  of  turning  verbal  information  into  visual  communications  involves  working 
with  space,  text,  fonts,  and  visual  relationships,  it  is  best  to  use  a  simple  bold 
“Sans  Serif”  fonts  (p.  139).  The  bold  type  makes  the  letters  sharper.  Allessi  and 
Trollip  (1991)  and  Williams,  Paprock,  and  Covington  (1999)  believe  it  is  best  to 
use  a  mixture  of  capital  and  lowercase  letters,  and  avoid  using  all  capital  letters. 

For  best  readability,  Williams,  Paprock,  and  Covington  (1999) 
recommend  using  a  maximum  of  seven  lines  in  height  and  seven  words  in  length 
per  line  on  a  given  presentation  page  (p.  139).  Using  these  guidelines  forces 
educators  to  be  brief  and  concise,  a  trait  often  expected  of  their  students. 
According  to  Allessi  and  Trollip  (1991),  all  CBT  presentations  should  say  just 
enough  to  teach  what  is  desired  (p.  43). 

The  quality  of  text  is  an  important  concept.  Not  only  must  the  font 
be  readable  and  the  text  brief,  but  it  must  precisely  convey  the  main  ideas  of  the 
lesson  so  that  the  learning  objectives  will  be  met.  Pronouns  with  unclear 
referents  should  not  be  used.  Additionally,  more  than  one  word  should  not  be 
used  to  refer  to  the  same  thing  and  the  use  of  the  same  word  at  different  times  to 
mean  different  things  should  be  avoided  (Allessi  and  T rollip,  1 991 ,  p.  43). 

It  should  go  without  saying  that  CBT  presentations  should  be  inherently 
flawless  with  respect  to  spelling,  grammar,  and  punctuation.  This  is  not  always 
an  easy  task  considering  that  brevity  is  also  considered  a  necessity.  The  key  to 
enhancing  student  learning  and  comprehension  is  consistency  throughout  the 
course  of  instruction. 

d.  Mid -Topic  Questioning  Techniques 

According  to  Allessi  and  Trollip  (1991)  and  Gronlund  (1998), 
questions  asked  during  a  lesson  serve  many  functions.  Mid-topic  questions  help 
keep  the  student  attentive,  provide  needed  practice,  encourage  deeper 
processing,  assess  student  recall  and  understanding,  and  provide  a  basis  for 
lesson  sequencing.  Each  of  these  is  an  appropriate  reason  to  include 
questioning  within  the  framework  of  each  topic.  I  recommend  asking  questions 
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about  material  about  to  be  taught  as  well  as  material  that  has  already  been 
taught.  Asking  questions  about  material  yet  untaught  will  focus  the  student’s 
attention  on  the  material  he  or  she  is  about  to  learn.  Questioning  students  about 
material  already  presented  will  assist  in  verifying  the  learners’  comprehension. 

It  can  be  difficult  to  know  how  much  questioning  should  be  used 
during  a  lesson.  Allessi  and  Trollip  (1991)  argue  that  it  should  occur  frequently 
because  the  more  a  student  interacts  with  the  program,  the  more  attention  will  be 
maintained  and  more  learning  will  be  facilitated.  Although  I  agree  frequent 
questioning  will  improve  attention,  I  believe  too  much  questioning  can  lead  to 
student  frustration.  Too  many  questions  during  a  lesson  may  be  interpreted  as 
unnecessary  interruptions  that  distract  and  anger  students.  I  recommend  using 
caution  when  placing  questions  in  the  middle  of  a  presentation.  Having  been  an 
enlisted  naval  instructor  for  nearly  four  years,  I  have  come  to  believe  it  is  better  to 
ask  a  few  questions  during  the  lesson  to  maintain  the  students’  attention  and 
then  ask  deeper  thought  provoking  questions  at  the  end  of  each  topic.  This 
technique  worked  very  well  for  me  in  the  classroom,  and  I  believe  it  would  work 
just  as  well  in  a  CBT  environment. 

6.  Student  Testing 

Assessing  student  performance  is  a  vital  aspect  of  all  training  programs. 
Assessments  can  be  summative  or  formative.  Summative  assessments  are 
completed  at  the  end  of  a  course  whereas  formative  assessments  are  used  to 
monitor  progress  throughout  the  course. 

a.  Testing  Schemas 

There  are  three  basic  testing  schemas.  In  the  first,  the  computer 
grades  students’  answers.  Horton  (2000)  argues  this  technique  has  two 
advantages.  Evaluation  is  immediate  and  the  computer  is  nonjudgmental  (p. 
277).  However,  Horton  (2000)  and  Allessi  and  Trollip  (1991)  report  that  because 
the  computer  is  limited  in  the  types  of  answers  it  can  grade,  teachers  can  only 
ask  simple  forms  of  questions.  The  second  schema  utilizes  the  student’s 
coworkers  or  managers  to  grade  exams.  According  to  Horton  (2000),  the 
disadvantage  to  this  approach  is  coworkers  may  not  be  available  or  may  not 
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have  the  expertise  required  to  evaluate  the  learner  (p.  277).  The  last  schema 
calls  for  the  student’s  answers  to  be  transmitted  to  the  instructor  for  grading. 
Because  instructors  can  spot  any  subtle  mistakes  and  grade  appropriately,  there 
is  no  limit  to  the  types  of  questions  that  can  be  asked  of  students  (Horton,  2000, 
p.  277). 

b.  Test  Construction 

Ciavarelli  (2003)  outlines  four  rules  to  follow  when  building  tests. 

They  are: 

(1)  Examination  content  must  be  based  upon  the  learning  objectives  (p.  14). 

(2)  Performance  requirements  for  the  examination  must  be  consistent  with  the 
learning  objectives  (p.  14) 

(3)  Examination  instructions  must  be  clear  (p.  14). 

(4)  Feedback  regarding  the  examination  must  be  timely,  constructive,  and 
relevant  (p.  14). 

c.  Remediation 

Remediation  is  the  follow-on  presentation  of  the  same  instructional 
material  to  a  student  who  did  not  meet  the  necessary  standards  on  an 
examination  or  performance  review.  Allessi  and  Trollip  (1991)  state  that 
remediation  can  occur  in  the  form  of  repeating  information  already  seen  or  the 
form  of  providing  information  in  more  detail  (p.  77).  This  can  include  giving  the 
student  more  examples,  pictures,  sample  problems,  or  practice  problems. 

C.  EVALUATIONS 

Evaluation  is  essential  to  any  educational  endeavor.  There  are  two 
fundamental  types  of  evaluations:  course  (or  program)  evaluation  and  student 
evaluation.  Both  are  necessary,  since  one  type  of  evaluation  examines  whether 
the  right  material  was  taught  to  the  students  in  an  effective  and  efficient  way,  and 
the  other  examines  whether  the  students  learned  the  material  sufficiently  well 
enough  to  conclude  the  learning  objectives  were  met.  As  can  be  seen, 
evaluations  determine  the  effectiveness  of  any  training  program. 
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Authors  that  write  about  evaluations  often  quote  Kirkpatrick.  Many 
consider  him  a  pioneer  in  the  field.  Kirkpatrick  (1998)  argues,  and  Dix  et  a!. 
(1997)  concur  that  there  are  three  primary  reasons  to  evaluate  training.  First,  to 
justify  the  existence  of  the  training  department  by  showing  how  much  it 
contributes  to  an  organization’s  objectives  and  goals.  Second,  evaluations  are 
used  to  decide  whether  to  continue  or  discontinue  a  training  program.  Third, 
evaluations  are  used  to  gain  information  on  how  to  improve  future  training 
programs. 

Whatever  type  of  evaluation  is  being  constructed,  it  needs  to  be 
contextually  relevant  (Moskal  and  Dziuban,  2001,  p.  161).  There  are  guidelines 
to  be  followed  to  ensure  evaluations  generate  valuable  and  useful  information  to 
be  used  to  make  the  training  program  better.  The  expense  of  time  and  money  to 
perform  an  evaluation  is  wasted  if  the  correct  information  is  not  being  collected. 

1.  Training  Objectives 

Nearly  half  a  century  ago,  Bloom  (1956)  identified  three  educational 
domains.  They  are  the  cognitive  domain,  the  affective  domain,  and  the 
psychomotor  domain  (p.  7).  Learning  objectives  within  the  cognitive  domain 
focus  on  recall  of  knowledge  and  the  development  of  intellectual  skill  (p.  7). 
Learning  objectives  within  the  affective  domain  focus  on  changes  in  interest, 
attitudes,  and  values  (p.  7).  Learning  objectives  in  the  psychomotor  domain 
focus  on  manipulative,  or  human  motor-skills  (p.  7). 

Traditionally,  student  performance  within  each  of  these  domains  is 
evaluated  differently.  Performance  in  the  cognitive  domain  is  usually  measured 
through  tests  and  papers.  In  a  traditional  college  classroom  this  equates  to  a 
midterm,  a  final  exam,  and  a  moderately  sized  research  or  opinion  paper.  It  is 
harder  to  evaluate  students  across  the  affective  domain  because  it  is  harder  to 
objectively  evaluate  a  student’s  attitudes.  Teachers  attempt  to  achieve  this  by 
taking  attendance  and  giving  a  grade  for  in-class  participation.  Performance 
based  tests,  or  demonstrations,  are  normally  used  to  evaluate  students’ 
psychomotor  skills. 
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a.  Bloom’s  Taxonomy 

Bloom  (1956)  developed  a  taxonomy,  or  method  of  classifying 
educational  objectives.  Bloom  (1956)  defined  an  educational  objective  as  the 
explicit  formulations  of  the  ways  in  which  students  are  expected  to  be  changed 
by  the  educational  process  (p.  26).  Each  category  that  is  presented,  builds  on 
the  previous  one.  A  student  can  be  thought  of  as  moving  through  these 
categories,  from  one  to  another,  as  knowledge  and  intellectual  skill  improve. 
Additionally,  the  way  questions  are  worded  when  assessing  student  achievement 
varies  from  one  objective  category  to  the  next.  Bloom’s  (1956)  taxonomy  of 
objectives  includes  the  following  categories:  knowledge,  comprehension, 
application,  analysis,  synthesis,  and  evaluation. 

Knowledge  is  remembering,  by  recall  or  reorganization,  some  idea 
or  phenomenon  with  which  the  student  has  had  experience  in  the  educational 
process  and  it  is  basic  to  all  other  ends  or  purposes  of  education  (Bloom,  1956, 
p.  18).  Basic  questions  that  test  recall  are  often  used  to  assess  knowledge.  For 

example,  “A  destroyer  is  a  type  of _ ?”  would  be  an  appropriate  question  if 

testing  in  the  knowledge  domain. 

Comprehension  is  related  to  behaviors  or  responses  that  represent 
an  understanding  of  the  literal  message  contained  in  a  communication  (Bloom, 
1956,  p.  89).  Asking  students  to  translate  or  interpret  ideas  are  effective  means 
to  assess  student  comprehension.  A  question  asking  students  to  illustrate  the 
concepts  depicted  on  a  chart  or  other  graphic  would  be  an  effective  way  to  test 
comprehension. 

Bloom  (1956)  defines  application  as  the  illustration  of  a  task  or  idea 
that  requires  the  comprehension  of  a  method,  theory,  principle,  or  applied 
abstraction  (p.  120).  Asking  a  student  to  explain  in  detail  the  underlying  reasons 
why  a  modern  ship  ran  aground  after  providing  the  student  with  a  set  of  facts 
about  the  case  in  question  is  an  example  of  how  to  assess  student  capabilities 
within  this  domain. 
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The  category  of  analysis  is  similar  to  application.  Analysis  is  the 
breaking  down  of  the  given  material  into  its  constituent  parts  and  detection  of  the 
relationships  of  the  parts  and  the  way  they  are  organized  (Bloom,  1956,  p.  144). 
Giving  students  an  article  to  read  and  then  asking  questions  about  the  article, 
like,  “Which  of  the  following  best  describes...”  or  “The  argument  that  was 
presented  was  flawed  because...”  is  effective  at  measuring  students’  analytical 
proficiency. 

Bloom  (1956)  describes  synthesis  as  the  putting  together  of 
elements  and  parts  so  as  to  make  a  whole  (p.  162).  He  argues  that  it  is  a 
combining  parts  and  elements  in  such  a  way  as  to  constitute  a  pattern  or 
structure  not  clearly  there  before  (p.  162).  Conducting  research  and  then 
developing  a  template  for  analyzing  CBT  could  be  considered  an  exercise  in 
synthesis.  Requiring  students  to  set  a  poem  to  music,  write  a  simple  melodic 
line,  or  write  a  composition  with  a  single  tonal  base  are  all  examples  of  testing 
students  within  the  synthesis  domain  (Bloom,  1956,  p.  179). 

Bloom’s  last  objective  is  evaluation.  Bloom  (1956)  defines 
evaluation  as  making  judgments  about  the  value,  for  some  purpose,  of  ideas, 
works,  solutions,  methods,  or  materials  (p.  185).  According  to  Bloom  (1956), 
evaluation  involves  the  use  of  quantitative  and  or  qualitative  criteria  as  well  as 
standards  for  appraising  the  extent  to  which  particulars  are  accurate,  effective, 
economical,  or  satisfying  (p.  185). 

These  domains  are  as  important  today  as  they  were  fifty  years  ago. 
With  respect  to  training,  organizations  know  to  what  extent  they  want  an 
employee  to  be  trained  in  a  specific  job.  Some  employees  only  need  have  a 
simple  understanding,  or  comprehension,  of  their  job  tasks.  A  gas  station 
attendant,  for  example,  may  only  need  to  know  how  to  pump  gas,  check  the  oil, 
and  clean  automobile  windows.  Other  employees  need  to  be  able  to  assimilate 
obscure  and  seemingly  unrelated  information  and  make  a  decision  that  may 
impact  the  organization  for  years  to  come.  It  is  likely  that  these  employees  will 
receive  training  that  will  enhance  their  ability  to  function  in  the  higher  (synthesis 
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and  evaluation)  domains.  It  is  important  that  organizations  build  mechanisms 
into  their  courses  of  instruction  that  verify  (1)  that  the  course’s  learning,  or 
educational,  objectives  are  in  agreement  with  organizational  goals,  and  (2)  that 
the  students’  performance  after  training  exceeds  the  standards  set  by  the 
educational  objectives. 

This  is  not  an  easily  accomplishable  task.  Kirkpatrick  (1998) 
contends  that  evaluating  the  results  of  training  is  very  difficult  (p.  60).  “What  is 
the  amount  that  quality  or  productivity  improved  as  a  result  of  training?”  or,  “What 
is  the  amount  training  contributed  to  profits?”  are  questions  that  many 
organizations  have  historically  been  unable  to  answer  (Kirkpatrick,  yr,  p.  60). 

However,  this  does  not  mean  that  it  is  impossible  to  validate 
training  objectives.  One  method,  although  time  consuming  and  rigorous,  that  has 
been  proven  successful  by  Housel  and  Bell  (2001)  in  answering  many 
Kirkpatrick’s  questions.  It  is  the  Knowledge  Value  Added  (KVA)  method. 
Organizations  that  properly  utilize  KVA  are  able  to  determine  if  their  training,  and 
specifically,  what  training  is  contributing  to  improved  or  decreased  revenues. 
Although  very  important,  it  should  be  noted  that  this  is  just  one  dimension  of 
course  or  curriculum  evaluation. 

2.  Evaluation  Design 

According  to  Bloom  (1956)  and  Renshaw  and  Taylor  (2000),  the  design 
and  evaluation  of  computer-based  instructional  materials  should  be 
accomplished  with  appropriate  memory  models  and  learning  theories  as  a 
reference.  Despite  the  importance  of  and  vital  need  to  conducting  evaluations, 
there  is  no  universally  agreed  upon  standards  or  methods  for  their  performance. 
Some,  like  Wiggins  (1998)  seem  to  advocate  more  of  an  emphasis  on  student 
evaluations,  while  others  like  Kirkpatrick  (1998)  seem  to  focus  on  program 
evaluations.  Despite  the  differences  in  their  approaches,  the  goal  remains  the 
same:  ensuring  that  the  learning  objectives  are  met. 

Hall  (1998)  suggests  that  three  themes  should  guide  the  development  of 
any  evaluation  model.  First,  Hall  (1998)  contends  that  an  overriding  direction, 
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taking  into  account  learners,  contexts,  and  goals,  should  be  meticulously 
identified  as  a  first  step  (p.  138).  Next,  Hall  (1998)  argues  that  proper  design  is  a 
matter  of  finding  the  proper  balance  between  elements  of  simplicity  and 
complexity  (p.  138).  In  other  words,  do  not  overdo  it.  Last,  Hall  (1998)  states 
that  evaluations  and  assessments  must  be  part  of  any  program  design  process 
because  without  evaluations,  the  designer  will  never  become  aware  of  how 
effective  a  training  program  is  (p.  138). 

Wiggins  (1998)  offers  six  criteria  that  should  be  examined  during  an 
evaluation.  According  to  Wiggins  (1 998),  all  assessments  should  be: 

(a)  Credible  to  all  stakeholders  (p.  Ill ). 

(b)  Useful,  helpful  to  teachers,  students,  and  administrators  (p.  111). 

(c)  Balanced  in  the  use  of  assessment  methods,  to  provide  a  rich, 
defensible,  and  feasible  profile  of  achievement  (p.  Ill ). 

(d)  Honest,  yet  fair  (p.  1 1 1 ). 

(e)  Intellectually  rigorous  and  thought  provoking;  focused  on  core  ideas, 
questions,  problems,  and  knowledge  (p.  111). 

(f)  Feasible  in  terms  of  resources,  logistics,  politics,  and  redeployment  of 
time  for  collaboratively  designing,  debugging,  using,  evaluating,  and  effectively 
reporting  student  work  (p.  111). 

Compare  these  student-focused  criteria  against  a  set  of  program-centered 
criteria.  Kirkpatrick  (1 998)  contends  that  there  are  eight  broad  factors  that  should 
be  considered  when  performing  an  evaluation  of  a  program.  They  are: 

(a)  To  what  extent  does  the  subject  content  meet  the  needs  of  those 
attending  (p.  17)? 

(b)  Is  the  instructor  the  one  best  q  ualified  to  teach  (p.  1 7)? 

(c)  Does  the  leader  use  the  most  effective  methods  for  maintaining 
interest  and  teaching  the  desired  attitudes,  knowledge,  and  skills  (p.  17)? 

(d)  Are  the  facilities  satisfactory  (p.  17)? 

51 


(e)  Is  the  schedule  appropriate  for  the  participants  (p.  1 7)? 

(f)  Are  the  aids  effective  in  improving  communication  and  maintaining 
interest  (p.  17)? 

(g)  Was  the  coordination  of  the  program  satisfactory  (p.  17)? 

(h)  What  else  can  be  done  to  improve  the  program  (p.  1 7)? 

It  is  obvious  that  each  of  the  previous  two  lists  were  designed  to  evaluate 
different  aspects  of  a  course  of  instruction.  The  objectives  in  each  case, 
however,  were  the  same,  to  ensure  that  the  students  learned  what  the  instructors 
intended  them  to.  To  argue  for  the  primary  use  of  one  of  these  two  approaches 
over  the  other  is  counterproductive.  Both  types  of  evaluations  are  essential  to 
ensuring  not  only  that  the  students  are  learning,  but  that  they  are  learning  the 
right  material.  I  agree  with  Belanger  and  Jordon’s  (2000)  argument  that  the  key 
consideration  for  the  success  of  any  distance-learning  environment  is  whether  or 
not,  and  to  what  degree,  the  learning  objectives  are  being  met  (p.  14). 

3.  Pedagogical  Evaluation 

There  are  many  evaluation  methods,  or  models,  available  for  use.  Some 
organizations  hire  other  firms  to  conduct  training  evaluations,  while  others 
develop  a  set  of  in-house  guidelines.  Some  approaches  to  evaluation  are 
simple,  and  some  are  very  complex.  Some  companies  even  prefer  to  use  a 
multifaceted  approach.  What  is  clear  is  that  the  approaches,  methods, 
evaluators,  and  scoring  rubrics  vary  greatly. 

Belanger  and  Jordon  (2000)  report  that  determinants  of  program  success 
can  be  grouped  into  the  four  categories  of  institutional  characteristics,  learner 
characteristics,  course  characteristics,  and  distance  learning  characteristics  (p. 
187).  Institutional  characteristics  factors  include  institutional  objectives,  delivery 
mechanisms,  and  support  structures  (Belanger  and  Jordon,  2000,  p.  1 87). 

Learner  characteristics  include  the  factors  of  motivation,  desire  to  excel, 
and  self-sufficiency  (Belanger  and  Jordon,  2000,  p.  188).  The  course 
characteristic  factors  include  group  project,  final  paper,  knowledge  based  or 
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performance-based  criterion  (Belanger  and  Jordon,  2000,  p.  188).  Distance 
learning  factors  consist  of  the  type  of  technology  and  learning  environment. 

This  structure  provides  a  decent  framework  for  building  a  comprehensive 
evaluation  template  from.  This  model  would  be  very  useful  in  making  baseline 
predictions  about  the  future  success  or  failure  of  a  training  program.  Upon 
examination,  this  model  does  not  define  any  of  the  factors  that  fall  into  the  four 
categories.  Just  as  it  is  important  that  organizations  adequately  define  their 
educational  objectives,  it  is  also  important  that  organizations  precisely  define  the 
measurement  categories,  factors,  and  rubrics  that  will  be  used  to  evaluate  the 
training  program. 

Parikh  and  Sameer  (2002)  describe  an  empirical  model  for  program 
evaluation  that  is  similar  to  Belanger  and  Jordon’s.  This  model  also  has  four 
dimensions.  Parikh  and  Sameer  (2002)  list  them  as  content,  technology, 
interface,  and  functionality  (p.  35).  The  content  section  of  this  model  measures 
the  characteristics  of  the  information  provided  by  the  system  to  the  users,  how 
precise  the  information  is,  what  is  its  value,  and  if  it  meets  the  users’  needs 
(Parikh  and  Sameer,  2002,  p.  35).  The  technology  category  is  used  to 
determine  whether  the  organization  is  utilizing  appropriate  technologies  to  deliver 
the  right  information  at  the  right  time  (Parikh  and  Sameer,  2002,  p.  35).  Interface 
examines  whether  the  system  is  easy  to  learn  and  user  friendly  (Parikh  and 
Sameer,  2002,  p.  35).  Parikh  and  Sameer  (2002)  state  that  evaluators  would 
use  this  section  to  verify  that  the  screen  layout  is  appropriate,  instructions  clear, 
and  information  is  presented  in  a  useful  format  (p.  35).  Functionality  measures 
the  practicality  as  opposed  to  the  attractiveness  of  the  system  (Parikh  and 
Sameer,  2002,  p.  35). 

4.  Instructional  Evaluation 

All  instructional  activities  need  to  be  assessed.  It  is  possible  students  are 
being  taught  old,  or  outdated  materials.  Unfortunately,  it  is  possible  that  an 
instructor,  even  though  an  expert,  may  not  be  very  good  at  teaching.  Tests  can 
be  biased  toward  a  particular  race  or  gender.  If  organizations  did  not  evaluate 
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the  various  aspects  to  instruction,  these  potential  problem  areas  might  never  be 
discovered.  If  never  discovered,  they  cannot  be  corrected. 

Gronlund  (1998)  believes  assessments  allow  organizations  to  evaluate  the 
effectiveness  of  three  parts  of  the  instructional  process.  Organizations  can 
ascertain  to  what  extent  instructional  objectives  were  realistic,  evaluate  whether 
methods  and  materials  of  instruction  were  appropriate,  and  determine  how  well 
the  learning  experiences  were  sequenced  (p.  11).  Gronlund  (1998)  offers  a  set 
of  guidelines  to  follow  when  performing  instructional  assessments  and  analyzing 
the  results: 

(1)  Organizations  must  have  a  clear  conception  of  all  intended  learning 
outcomes  (p.  18).  What  types  of  knowledge,  understanding  and  application  does 
the  organization  want  students  to  gain  and  what  evidence  is  the  organization 
willing  to  accept  as  proof  that  the  students  have  achieved  to  that  level  (p.  18). 

(2)  The  organization  should  use  a  variety  of  assessment  procedures  to 
verify  student  achievement  (p.  18).  These  can  include  tests,  quizzes,  surveys, 
performance  assessments  of  teachers  and  students,  or  a  combination  of  these 
methods  (p.  18). 

(3)  The  instructional  relevance  of  the  procedures  must  be  considered  (p. 
18).  Consider  what  happens  if  a  student  completes  a  CBT  lesson,  but  the 
instructional  material  only  covered  three  of  the  five  learning  objectives.  Gronlund 
(1998)  argues,  and  I  agree,  the  instruction  has  lost  relevance  because  the 
learning  objectives,  instruction,  learning  tasks,  and  assessment  procedures  are 
not  in  close  agreement.  There  is  hardly  a  greater  “instructional  sin”  than  telling 
students  that  they  will  be  required  to  learn  material  to  a  organizationally  defined 
standard  and  then  not  allowing  them  to  meet  that  standard  because  not  enough 
information,  or  the  wrong  information,  was  presented. 

(4)  Specifications  of  criteria  for  judging  successful  performance  must  be 
known  and  published  (p.  20).  Without  clearly  defined  and  published  grading 
criteria,  it  is  easy  to  demotivate  students.  For  example,  assume  a  class  of 
students  is  required  to  complete  an  essay,  but  they  have  not  been  told  what  the 
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grading  criterion  is  going  to  be.  When  the  papers  are  handed  back,  the  students 
that  did  not  receive  the  highest  grades  will  not  only  wonder  if  the  papers  were 
graded  fairly,  but  will  also  lose  motivation  and  interest  because  they  are  unsure 
what  how  to  write  an  “A”  paper  next  time. 

(5)  Feedback  must  emphasize  the  strengths  of  performance  and  any 
weaknesses  that  need  to  be  corrected  (p.  21). 

(6)  A  comprehensive  grading  and  reporting  system  must  be  in  place  (p. 
21).  Gronlund  (1998)  infers  that  letter  grades  should  be  eliminated  (p.  21). 
Letter  grades  help  educational  institutions  separate  good  performers  from  poor 
ones,  and  have  value  in  public  education.  However,  in  an  organizational  training 
environment,  letter  grades  mean  very  little.  Employees  would  be  better  served  if 
they  were  provided  specific,  objective,  relevant,  and  non-biased  feedback  that 
praised  their  strengths,  identified  their  weaknesses,  and  concluded  by  offering 
suggestions  for  improving  performance.  Furthermore,  it  this  feedback  would  be 
very  valuable  if  it  were  given  many  times  throughout  a  lengthy  course  of 
instruction,  as  opposed  to  once  at  the  end  of  the  class. 

D.  COMPREHENSIVE  MODEL  FOR  THE  EVALUATION  OF  CBT 

This  CBT  evaluation  framework  consists  of  two  sections.  This  first  section 
is  Pedagogical  Evaluation.  The  second  section  is  Instructional  Evaluation.  Each 
section  consists  of  four  functions.  The  Pedagogical  Evaluation  section  contains 
a  Program  Operation  function,  an  Instructional  Presentation  function,  an 
Assessment  Function  and  a  Feedback  function.  The  Instructional  Evaluation 
section  consists  of  a  Course  and  Learning  Objectives  function,  a  Core 
Curriculum  function,  along  with  Assessment  and  Feedback  functions.  Figure  4 
displays  a  representation  of  this  evaluative  framework.  Two  sets  of 
questionnaires  were  developed  for  each  section.  One  set  is  designed  for 
developers  and  program  evaluators  to  use,  the  other  set  is  designed  to  gather 
information  from  students.  Appendix  A  contains  all  of  the  questionnaires. 
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Figure  4.  CBT  Evaluative  Framework 
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Two  elements,  Interactivity  and  Instructional  Sequence,  overlap  the 
boundaries  of  the  two  main  sections.  These  elements  fit  best  within  the 
Pedagogical  Evaluation  section’s  Instructional  Presentation  function,  but  cannot 
be  properly  evaluated  without  taking  into  account  the  Educational  Objectives 
function’s  learning  objectives  and  the  instructional  content  of  the  Core 
Curriculum. 

The  Instructional  Quality  element  assesses  the  overall  quality  and 
effectiveness  of  the  CBT  initiative.  The  strengths  and  weaknesses  of  the 
Pedagogical  Evaluation  and  Instructional  Evaluation  sections  are  compared  and 
the  instructional  effectiveness  of  the  CBT  initiative  is  determined.  In  other  words, 
it  is  a  gage  of  how  well  the  CBT  courseware  meets  organizational  training 
objectives  through  the  proper  presentation,  monitoring,  and  assessment  of 
instructional  content,  and  how  well  feedback  is  provided  to  both  the  organization 
and  its  students. 

1.  Pedagogical  Evaluation. 

This  section  of  the  evaluation  framework  will  assess  the  various  aspects  of 
CBT  that  are  not  associated  with  the  information  that  is  taught  and  the  way 
information  is  tested  throughout  the  student  population.  The  information 
gathered  using  the  functions  in  this  section  will  enable  organizational  leadership 
and  those  who  develop  CBT  programs  to  effectively  identify  the  strengths  and 
weaknesses  inherent  in  the  techniques  their  CBT  software  programs’  are 
installed,  used,  and  presented  to  students.  Once  deficiencies  are  identified, 
organizations  can,  based  upon  their  needs  or  budgetary  constraints,  implement 
procedures  that  will  eliminate  or  minimize  the  identified  weaknesses. 
a.  Program  Operation 

The  Program  Operation  function  examines  the  software’s  interface 
and  amount  of  control  that  each  learner  has  while  using  the  tutorial. 

1.  Interface.  The  interface  between  the  student  and  the 

computer  is  very  important.  It  determines  what  the  student  is  going  to  look  at  on 

the  computer  screen  while  navigating  through  each  lesson.  This  element 
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critically  examines  screen  organization,  student  control  during  training,  ease  of 
installation  and  ease  of  use. 

2.  Security.  This  element  assesses  whether  students’ 
personal  information,  test  scores,  or  performance  evaluations  are  protected  from 
intrusion,  tampering,  or  theft. 

b.  Instructional  Presentation 

The  Instructional  presentation  function  assesses  the  quality  of  CBT 
instruction.  This  function  will  not  evaluate  the  instructional  content  of  a  lesson; 
only  how  well  it  is  presented. 

1.  Interactivity.  This  element  evaluates  whether  a  CBT 
lesson  is  interactive.  Although  the  objective  of  this  element  is  not  to  evaluate 
instructional  content,  whether  a  lesson  contains  an  appropriate  amount  of 
interactivity  is  determined  in  a  large  part  by  the  instructional  content’s  length  and 
complexity. 

2.  Lesson  Sequencing.  The  lesson-sequencing  portion  of 
this  section  assesses  whether  or  not  lesson  topics  are  properly  sequenced. 
Although  the  objective  of  this  element  is  not  to  evaluate  instructional  content, 
whether  a  series  of  modules  or  lessons  are  correctly  sequenced  is  largely 
determined  by  lesson  content  and  the  principle  of  teaching  from  simple  to 
complex. 

3.  Multimedia  Format.  This  element  evaluates  whether 
incorporated  graphics,  animations,  cartoons,  video,  and  audio,  is  being  used 
correctly. 

4.  Color.  This  element  evaluates  whether  the  use  of  color 
has  been  implemented  correctly. 

5.  Text.  This  element  verifies  that  text  has  been  properly 
laid  out  and  spaced,  and  that  the  presentation  is  grammatically  correct  at  an 
appropriate  reading  level. 

6.  Questioning.  This  element  assesses  whether  appropriate 
questioning  techniques  have  been  implemented  throughout  the  lesson. 
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7.  Environment.  This  element  analyzes  the  suitability  of  the 
students’  learning  environment  while  using  CBT. 

c.  Assessment 

This  function  of  the  pedagogical  evaluation  section  evaluates  how 
the  organization  conducts  assessments. 

1.  Methods.  This  element  evaluates  whether  organizational 
assessment  methods  are  defined  and  used. 

2.  Rubrics.  In  this  instance,  a  rubric  is  the  standard  with 
which  each  function  of  the  CBT  program  will  be  evaluated.  This  element 
evaluates  whether  rubrics  for  each  “area”  the  organization  will  assess  have  been 
properly  defined. 

3.  Participants.  This  element  determines  if  all  of  the 
evaluation  participants,  along  with  each  one’s  role  in  the  evaluation  has  been 
defined  and  promulgated. 

4.  Usefulness.  This  element  assists  in  evaluating  whether  or 
not  the  information  collected  during  an  assessment  is  useful  to  the  organization. 

d.  Feedback 

Feedback  regarding  the  pedagogical  effectiveness  of  any 
organizational  training  initiative  is  critical  for  the  success  and  continued 
improvement  of  that  venture.  This  function  evaluates  whether  the  feedback  to 
the  organization  regarding  their  CBT’s  pedagogical  effectiveness  is  timely, 
relevant,  and  useful. 

2.  Curriculum  Evaluation 

The  Curriculum  Evaluation  section  of  this  model  will  analyze  the  course 
content  for  relevancy,  correctness,  and  usefulness  to  both  the  organization  and 
those  undergoing  the  training. 

a.  Learning  Objectives 

This  element  of  the  model  will  verify  several  things.  First,  that  the 
organizational  goals  with  respect  to  a  particular  training  program  are  defined. 
Second,  it  will  determine  if  the  training  program’s  learning  objectives  have  been 
derived  from  the  program’s  stated  goals  and  if  the  students’  performance 
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standards  were  derived  from  the  learning  objectives.  Third,  it  will  verify  that  the 
learning  objectives  are  presented  in  the  CBT  program  in  terms  of  student 
performance. 

b.  Core  Curriculum 

This  function  will  verify  that  the  instruction  that  the  students  receive 
is  relevant,  current,  and  useful  to  the  students.  It  will  also  verify  that  as  a  result 
of  this  instruction,  students  are  able  to  meet  or  exceed  all  required  performance 
standards. 

c.  Assessment 

This  function  of  the  pedagogical  evaluation  section  evaluates  how 
the  organization  conducts  assessments. 

1.  Methods.  This  segment  evaluates  whether  assessment 
methods  for  each  lesson  are  defined  and  used. 

2.  Rubrics.  This  segment  evaluates  whether  rubrics  for  each 

test  or  performance  assessment  have  been  properly  defined. 

3.  Participants.  This  segment  determines  if  all  of  the 
participants,  along  with  each  one’s  role  in  the  test,  quiz,  or  performance 
assessment  has  been  defined  and  promulgated. 

4.  Usefulness.  This  segment  assists  in  determining  whether 
or  not  the  information  collected  tests  and  performance  evaluations  are  useful  to 
the  organization. 

d.  Feedback 

Feedback  regarding  the  instructional  effectiveness  of  any 
organizational  training  initiative  is  essential  for  the  success  and  continued 
improvement  of  that  venture.  This  function  evaluates  whether  the  feedback  to 
the  organization  and  the  students  regarding  the  CBT  instruction  and  testing  is 
timely,  relevant,  and  useful. 

3.  Conclusion 

Use  of  this  template  and  its  associated  survey  sheets  found  in  Appendix  A 
will  allow  evaluators  to  conduct  comprehensive  assessments  of  their  CBT 
programs.  The  survey  sheets  will  be  used  to  conduct  an  evaluation  of  the 
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reengineered  CBT  Surface  Warfare  Officer  Division  Officer  course  of  instruction. 
The  result  will  be  an  accurate  assessment  of  the  course’s  cognitive  and 
pedagogical  strengths  and  weaknesses. 

With  minor  modifications  to  the  pedagogical  evaluation  section  of 
template,  it  could  be  effectively  used  to  evaluate  the  strengths  and  weaknesses 
of  traditional  classroom  instruction.  Specifically,  computers  differ  from  humans  in 
the  way  in  which  they  present  information.  Since  they  present  instructional 
materials  differently,  different  criteria  for  conducting  that  portion  of  an  evaluation 
is  required. 
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IV.  ANALYSIS 


This  analysis  is  being  conducted  to  fulfill  two  objectives.  First,  it  acts  as  a 
proof  of  concept  that  the  evaluation  framework  presented  in  the  previous  chapter 
works.  Second,  it  depicts  an  accurate  snapshot  of  the  SWOS  Division  Officer  at 
Sea  Training  Initiative  at  the  present  time.  Since  its  inception  in  January  2003, 
very  few,  if  any  junior  officers  have  completed  this  training  initiative  and  earned 
their  Surface  Warfare  Pin.  This  is  to  be  expected  since  the  majority  of  newly 
accessioned  officers  just  joined  the  fleet  in  June  and  July  of  2003. 

A.  DATA  COLLECTION  METHOD 

1.  Participants 

Officers  stationed  aboard  ships  homeported  in  San  Diego,  California 
completed  the  surveys  in  August  2003.  17  Ensigns  completed  the  student 

questionnaire  and  four  Training  Officers  completed  the  supervisor  survey.  These 
officers  were  promised  complete  anonymity  in  exchange  for  their  honesty  in  filling 
out  the  surveys.  Consequently,  ship  types,  names,  and  hull  numbers  are  omitted 
from  this  report.  Small  group  discussions  were  held  one  each  ship  immediately 
after  the  surveys  were  completed. 

2.  Survey  Instruments 

Two  surveys  were  generated  to  assist  an  evaluator  assessing  a  CBT 
program  using  the  framework  developed  in  chapter  III.  One  survey  is  tailored  for 
course  administrators  and  one  for  students.  The  survey  questions  were 
generated  through  multiple  means.  Questions  were  developed  based  upon  the 
literature  review  and  the  principles  discussed  in  Chapters  II  and  III.  Dr.  Anthony 
Ciavarelli,  an  expert  in  the  field  of  organizational  training  who  teaches  at  the 
Naval  Postgraduate  School,  Monterey,  also  contributed  questions.  These 
questions  started  with  a  baseline  survey  assessing  the  quality  of  online 
instruction  derived  from  Ciavarelli  (2003).  Several  questions  are  worded  in  such 
a  way  as  to  only  be  applicable  to  students  learning  in  a  shipboard  environment, 
but  with  minor  modifications,  could  easily  be  asked  to  students  in  any  CBT 
learning  environment.  Each  of  the  two  surveys  had  three  sections.  Section  one 
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asks  basic  background  information.  Section  two  presents  multiple  statements 
and  asks  students  to  indicate  whether  they  agree  or  disagree  with  the  statement. 
A  Leikert  scale  from  one  to  five  is  used  in  this  section.  Section  three  asks 
students  to  provide  short  essay  answers  to  multiple  questions.  A  complete 
analysis  of  each  segment  in  the  Evaluation  Framework  will  not  be  provided. 
Instead,  areas  that  have  identified  strengths  or  weaknesses  will  be  discussed  in 
depth.  Complete  survey  data  is  provided  in  Appendix  B. 

B.  TRADITIONAL  CLASSROOM  SWO  TRAINING 

The  curriculum  that  was  used  in  the  classroom  to  train  Ensigns  designated 
to  become  Surface  Warfare  officers  was  converted,  in  its  entirety,  to  a  computer- 
based  training  program  that  is  now  loaded  onto  computers  from  compact  discs. 
In  order  to  effectively  determine  whether  the  new  teaching  method  is  as  effective 
as  the  one  it  replaced,  an  overview  of  the  strengths  and  weaknesses  of  the 
historical  classroom  training  program  is  necessary. 

Historically,  the  Surface  Warfare  Officers  School,  Division  Officer 
Curriculum  (SWOS  DOC)  made  extensive  use  of  the  classroom  environment  to 
train  newly  commissioned  Ensigns  who  were  to  be  permanently  assigned  to  the 
Navy’s  Surface  Warfare  community.  The  Surface  Warfare  classroom  curriculum 
structure  has  been  modified  several  times  over  the  years,  but  the  information 
taught  to  those  who  received  the  training  remained,  for  the  most  part,  constant. 

Instruction  typically  began  at  0800  and  usually  concluded  near  1600. 
Each  fifty  minutes  of  instruction  ended  with  a  ten-minute  break.  Students  could 
expect  to  receive  a  daily  lunch  break  that  would  last  between  thirty  minutes  and 
one  hour,  usually  between  1100  and  1200.  The  students’  daily  routines  were 
typical  for  most  naval  personnel,  both  officer  and  enlisted,  who  have  received 
formal  classroom  training  during  the  past  twenty  years. 

1.  Core  Phases  of  Instruction 

The  Surface  Warfare  Officer  School’s  website  states  the  Surface  Warfare 
curriculum  was  designed  to  prepare  newly  commissioned  Ensigns,  enroute  their 
first  tours  as  division  officers  afloat,  to  stand  in  port  and  underway  watches,  and 

to  manage  the  administrative  duties  of  the  division  officer  afloat.  The  curriculum 
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was  divided  into  three  phases:  Core  Phase  I,  Core  Phase  II,  and  Billet  Specialty 
Training.  Core  Phase  I  consisted  of  eleven  weeks  of  operations  and  combat 
systems  training,  Core  Phase  II  consisted  of  six  weeks  of  platform  specific 
engineering  training,  and  Billet  Specialty  Training  was  comprised  of  three  to  six 
weeks  of  tailored  operations,  combat  systems  and  engineering  training  specific  to 
the  trainee’s  prospective  shipboard  billet. 

The  Core  Phase  I  curriculum  was  divided  into  units.  These  units  were 
subdivided  into  modules,  which  were  then  broken  down  further  into  topics.  It  is 
the  topics  that  were  taught  in  the  classroom.  Including  tests,  quizzes,  and 
reviews;  the  Core  Phase  I  curriculum  had  186  topics  that  were  taught.  Appendix 
C  contains  a  listing  of  these  topics  taken  from  Intelligent  Design  System’s,  Inc.’s 
(IDSI)  Final  Report. 

Core  Phase  II  was  divided  into  six  units.  Course  content  varied; 
depending  on  the  class  of  ship  the  student  will  be  going  to.  For  example,  a 
student  with  orders  to  an  Oliver  Flazard  Perry  class  frigate  (FFG)  would  receive 
instruction  that  covers  the  basics  of  a  gas  turbine  propulsion  plant  and  a  student 
with  orders  to  an  Austin  class  Landing  Platform  Dock  (LPD)  would  receive 
instruction  in  basic  steam  propulsion. 

Billet  Specialty  Training  was  comprised  of  one  or  more  service  schools 
that  provided  the  junior  officer  with  the  basic  information  that  he  or  she  must 
know  in  order  to  satisfactorily  perform  their  assigned  jobs  once  they  arrive 
aboard  their  first  ship.  For  example,  a  prospective  frigate  Combat  Information 
Center  Officer  (CICO)  is  usually  also  the  ship’s  Electronic  Warfare  Officer 
(EWO).  This  officer  would  first  receive  a  week’s  training  in  CICO  School, 
followed  by  a  week  of  training  in  EWO  School.  At  the  completion  of  this  training 
it  is  assumed  that  the  Ensign  possesses  enough  basic  knowledge  about  the 
duties  and  responsibilities  of  the  job  that  he  or  she  will  be  able  to  immediately 
perform  adequately  in  that  role  with  minimal  supervision. 

It  is  important  to  remember  that  although  it  is  difficult  to  remember  all  of 
the  information  presented  in  a  six-month  course  of  study,  the  information  that  this 
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curriculum  imparted  is  important  because  it  sets  the  knowledge  foundation  on 
which  the  new  officer  will  build  over  the  course  of  a  military  career.  Once  newly 
commissioned  officers  reported  aboard  their  first  ships,  further  training  drew  out 
and  reinforced  what  was  learned  at  SWOS.  Additionally,  SWOS  presented 
newly  commissioned  officers  with  all  of  the  information  required  by  the  Personnel 
Qualification  Standards  (PQS)  to  successfully  qualify  to  stand  the  two  watch 
stations  that  are  needed  to  earn  the  Surface  Warfare  Pin:  Combat  Information 
Center  Watch  Officer  and  Officer  of  the  Deck  Underway.  SWOS,  however,  did 
not  provide  the  practical  experiences  necessary  to  adequately  apply  this 
knowledge,  and  subsequently  qualify  to  stand  these  watches. 

2.  Test  Demographics 

Approximately  ten  years  of  historical  tests  scores,  forty  classes  and  over 
5000  students,  were  analyzed  and  the  outcomes  consolidated.  Although 
summarized  here,  appendix  C  contains  the  detailed  tabulated  results.  On 
average,  each  class  contained  183  students.  Students  were  required  to  achieve 
relatively  high  scores  to  pass  an  exam.  To  pass  the  Rules  of  the  Road  test, 
students  must  have  scored  at  least  a  ninety  percent.  Eighty  percent  was 
required  to  successfully  pass  all  other  exams  and  quizzes.  Historically,  the 
Navigation  module  proved  to  be  the  toughest  to  successfully  pass.  This  module 
had  two  tests.  On  average,  33  students  (18  percent)  failed  the  first  exam  and  38 
students  (21  percent)  failed  the  second  one.  This  module  also  contained  a 
graded  practical  that  roughly  84  students  (46  percent)  in  each  class  failed. 
Students  also  had  historical  difficulties  passing  the  Rules  of  the  Road  and 
Combat  Systems  tests.  Both  of  these  exams  had  historical  failure  rates  above 
ten  percent. 

As  it  turns  out,  unexpectedly,  when  asked  in  a  survey  by  IDSI  to  rate  the 
importance  of  each  of  the  topics  at  SWOS,  Surface  Warfare  officers  15  highest 
ranked  topics  are  contained  in  the  modules  that  have  the  highest  failure  rates 
among  students.  A  table  summarizing  the  results,  which  is  published  in  IDSI’s 
Final  Report  is  included  in  appendix  C  subsection  A. 
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3.  Effectiveness  of  the  Curriculum 

Research  shows  that  in  many  instances,  students  view  their  teachers,  not 
their  textbooks,  as  the  primary  source  of  information.  This  is  true,  in  part,  due  to 
the  fact  that  students  believe  that  they  can  rely  on  their  teacher  to  develop  new 
concepts  sufficiently  well  that  the  associated  test  can  be  passed  without  reading 
the  book  (Nix  and  Spiro,  1990,  p.  110).  It  is  possible  that  this  strategy  served 
students  well  enough  in  high  school  and  college,  but  it  may  be  inadequate  to 
facilitate  acceptable  learning  transfer  when  a  student  is  faced  with  internalizing 
extensive  quantities  information  that  is  passively  learned  over  a  shorter  periods 
of  time. 

Remediation  for  students  that  failed  an  exam  was  carried  out  through  a 
mandatory  study  program.  These  students  were  required  to  study  within  the 
schoolhouse  for  an  arbitrarily  designated  period  of  time  each  week,  typically  20 
hours.  This  studying  was  done  on  the  students’  time  after  normal  working  hours. 
This  process  was  used  by  the  school  as  a  necessary  motivational  tool  for 
improving  study  habits  and  ultimately,  retained  knowledge.  Since  the  school 
experienced  a  very  low  attrition  rate,  can  the  remediation  process  can  be 
classified  as  a  success.  This  approach  appears  to  have  helped  students 
prepare  for  and  pass  their  next  exam,  but  it  did  not  necessarily  enable  them  to 
successfully  use  that  knowledge,  or  several  skill  sets  taught  at  SWOS,  when  they 
first  arrived  aboard  their  ships. 

In  its  final  report  to  the  Chief  of  Naval  Education  and  Training,  Intelligent 

Decision  Systems,  Inc.  (IDSI)  posted  the  results  of  a  survey  that  attempted  to 

gain  insight  into  what  fleet  officers  thought  were  the  core  competencies  required 

to  be  a  successful  division  officer.  The  survey  asked,  “What  competencies  do 

you  believe  to  be  most  necessary  for  a  division  officer  to  be  effective?”  The 

survey  also  asked,  “From  your  experience,  what  competencies  are  most  often 

lacking  in  SWOS  graduates?”  22  of  55  (40  percent)  of  the  respondents  listed  as 

a  most  often  lacked  competency  at  least  one  of  the  items  that  they  listed  as  most 

necessary  for  division  officer  success.  Comments  included  such  things  as 

administrative  programs,  customs  and  courtesies,  naval  platforms  and  their 
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missions,  seamanship,  communications,  rules  of  the  nautical  road,  chain  of 
command,  technical  knowledge,  and  fundamental  knowledge  of  guns,  missiles, 
and  basic  engineering.  Comments  regarding  competency  deficiencies  in 
seamanship,  ship  handling,  and  watch  standing  were  common. 

If  only  IDSI  surveys  were  used  to  judge  this  course  of  instruction,  some 
could  arrive  at  the  erroneous  conclusion  that  the  program  was  not  successful 
because  Ensigns’  initial  performance  aboard  their  first  ships  was  lacking  in  some 
way.  The  IDSI  surveys  do  not  address  the  number  of  Ensigns  whose 
performance  the  commanding  officer’s  had  been  pleased  or  displeased  with.  It  is 
possible  that  when  answering  the  two  survey  questions  dscussed  above,  the 
respondents  had  only  seen  one  or  two  junior  officers  that  arrived  aboard  with  the 
deficiencies  listed.  It  is  also  possible,  that  the  respondents  were  answering  the 
surveys  generally,  and  found  most  Ensigns  lacking  the  knowledge  or  skills  that 
were  described  in  the  answers. 

Based  on  my  experiences,  there  may  be  some  merit  to  the  notion  that 
some  Ensigns  lacked  in  one  or  more  aspects  of  knowledge  or  specific  skills  when 
they  graduated  from  SWOS.  However,  even  if  true,  the  program  cannot  be 
described  as  a  failure.  It  is  possible,  that  utilizing  a  traditional  classroom-lecture 
approach,  coupled  with  minimal  laboratory  training  prior  to  sending  new  Ensigns 
to  their  first  ships,  resulted  in  an  inadequate  internalization  of  knowledge, 
insufficient  knowledge  transfer,  and  subsequent  deficits  in  either  knowledge  or  a 
particular  skill  set  that  would  lead  to  the  comments  found  in  the  above  mentioned 
survey.  It  is  also  possible,  and  argued  by  many  that  I  have  informally  talked  to, 
that  the  program  was  extremely  successful,  and  only  a  handful  of  Ensigns  ever 
reported  aboard  their  first  ship  deficient  in  either  knowledge  or  a  particular  set  of 
skills. 

Whatever  the  case,  as  a  rule,  once  Ensigns  checked  aboard  their  first  ship 
they  received  a  lot  of  on  the  job  training  (OJT)  and  spent  many  hours  studying  for 
watchstation  qualifications.  The  Ensigns  were  working  with  and  studying  the 
same  information  already  taught  to  them  in  SWOS.  It  is  likely  the  information 
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was  recalled  more  easily  since  it  had  already  been  seen  once  before,  processed, 
and  stored  in  memory  while  at  SWOS.  However,  there  may  have  not  been 
enough  time  to  rehearse  with  the  information  that  was  taught  in  SWOS  to 
successfully  build  semantic  mappings.  By  being  afforded  the  opportunity  to 
study  the  information  over  a  more  prolonged  period  of  time,  semantic  mappings 
were  quickly  constructed,  and  in  general,  the  junior  officers  rapidly  became 
knowledgeable  and  able  to  satisfactorily  perform  the  tasks  that  were  difficult 
when  they  first  checked  aboard. 

4.  Personal  Experience 

Students  had  opportunities  to  practice  their  budding  skills  in  numerous 
laboratories.  Students  were  able  to  practice  ship-handling,  communications,  and 
damage  control  procedures  in  a  relatively  benign  environment.  Since  student 
egos  were  the  only  things  ever  broken,  these  labs  proved  to  be  a  positive  training 
experience.  However,  time  in  these  trainers  was  limited,  and  the  best  that  could 
be  expected  from  a  novice  student  was  basic  familiarity  with  the  concepts  and 
principles  being  taught. 

5.  Conclusions  About  the  Traditional  SWOS  Curriculum 

The  traditional  SWOS  division  officer  curriculum  possessed  defined 
learning  objectives,  a  first-rate  core  curriculum,  a  feedback  mechanism  that 
effectively  served  its  purpose,  good  assessment  execution,  and  an  established 
remediation  process.  It  is  possible  that  since  large  quantities  of  information  was 
taught  very  rapidly,  that  some  Ensigns  could  not  process  and  rehearse  with  that 
information  long  enough  to  shift  it  into  long-term  memory  and  subsequently,  into 
a  semantic  mapping.  As  a  result,  some  of  the  Ensigns’  knowledge  within  the 
realm  of  nautical  instruction  remained  inert.  Many  times,  out  of  necessity, 
Ensigns  focused  on  and  memorized  enough  facts  to  pass  their  next  exam. 
These  facts,  however,  tended  to  be  easily  forgotten,  and  so  the  knowledge 
remained  “inert”  (Nix  and  Spiro,  1990,  p.  123). 

If  the  new  training  initiative  is  going  to  receive  good  reviews  from  the  fleet, 
an  instructional  methodology  is  needed  that  helps  the  students  use  multiple  facts 
in  constructive  problem  solving  environments  so  that  not  only  are  facts 
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remembered,  they  become  useful  tools.  Curriculums  that  utilize  delivery 
methods  that  incorporate  learning  activities  within  the  context  that  these  activities 
normally  occur,  along  with  a  presentation  of  the  culture  in  which  they  occur,  will 
best  achieve  the  goal  of  knowledge  transfer  (Brown,  Collins,  and  Duguid,  1989, 
P-33). 

C.  FIELD  OBSERVATIONS 

I  am  able  to  make  several  personal  observations  regarding  this  training 
initiative.  These  observations  are  in  addition  to  survey  data,  but  include 
information  learned  during  post-survey  discussions.  Collecting  survey  data  and 
making  self-reporting  field  observations  enables  the  viewing  of  information  from 
multiple  perspectives,  and  allows  the  verification  of  the  specific  field  observations 
and  survey  data,  to  what  extent  each  agrees  with  the  other.  Four  general 
observations  are  made.  First,  implementation  problems  exist  in  every  ship  on  the 
waterfront.  Second,  ship  Training  Officers  are,  with  respect  to  the  Division 
Officer  At  Sea  training  initiative,  very  frustrated.  Third,  the  students  are 
discouraged  about  the  Division  Officer  At  Sea  training  initiative.  Forth,  the 
training  initiative’s  two  stated  goals  are  not  well  defined  1 . 

1.  Implementation 

Every  ship  I  visited  was  currently  experiencing  implementation  problems. 
One  ship  had  not  begun  implementing  the  training  program.  Two  ships  had  only 
one  stand-alone  computer  that  was  used  for  this  CBT.  Several  ships  had 
installed  the  courseware  on  their  local  area  network  (LAN),  but  spent  months 
troubleshooting  bugs,  and  were  still  experiencing  problems  at  the  time  of  my  visit. 
None  of  the  training  officers  I  talked  to  reported  that  their  ship  was  not 
experiencing  difficulties  with  implementation. 

The  manner  in  which  this  initiative  has  been  locally  implemented  differs  in 
two  discernable  ways  than  was  discussed  at  the  first  kick-off  meeting  in  May 
2002.  At  this  meeting,  it  was  generally  agreed  that  the  Ensigns  were  not  to  be 
given  division  officer  responsibilities  until  after  they  had  qualified  Officer  of  the 

1  8  ships  were  visited.  Post  survey  discussions  with  training  officers  also  included  the 
implementation  problems  of  the  other  ships  along  the  waterfront. 
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Deck  (Underway).  This  qualification  has  historically  taken  between  12  and  18 
months  to  complete  depending  upon  the  officer’s  abilities  and  ship’s  schedule.  I 
observed  that  many  of  the  junior  officers  enrolled  in  this  program,  most  having 
been  attached  to  their  ships  for  less  than  three  months,  were  assigned  significant 
divisional  responsibilities.  Second,  there  was  after  a  lot  of  debate  in  this  kickoff 
meeting,  a  general  consensus  that  the  Ensigns  would  need  computing  resources 
supplied  to  them.  The  issue  at  the  meeting  was  whether  to  directly  supply  each 
Ensign  with  a  laptop  computer  or  supply  ships  with  computers  that  were  to  only 
be  used  in  conjunction  with  this  program.  To  date,  neither  has  occurred  and  the 
Ensigns  are  falling  behind  schedule  due  to  a  lack  of  computer  resources. 

2.  Training  Officers 

This  process  frustrates  Training  Officers.  Training  Officers  believe  that 
they  were  delivered  a  poor  beta  product  and  asked  to  implement  it  without 
sufficient  instructions,  training,  or  support  from  any  of  the  other  agencies  involved 
with  this  program. 

3.  Students 

Every  Ensign  I  talked  to  appeared  to  want  to  succeed.  Several,  however, 
felt  doubtful  about  their  future  success  in  this  program  because  of  the 
competition  for  resources  and  OJT  time  on  the  bridge.  The  students  are 
competing  for  resources.  Some  ships  have  8  to  10  Ensigns  working  their  way 
through  this  training  initiative.  Ships  do  not  have  enough  computers  to  study  at. 
There  are  not  enough  technical  manuals,  publications,  or  instructions  available  to 
simultaneously  support  shipboard  operations  and  student  learning.  Ensigns  are 
being  given  division  officer  responsibilities,  which  occupy  a  tremendous  amount 
of  their  time  and  significantly  reduces  the  time  available  for  quality  learning. 

OJT  time  is  reduced  for  each  Ensign.  Ships  are  underway  for  limited 
periods  of  time  each  quarter.  Ships  are  becoming  overmanned  with  junior 
officers,  and  as  a  result,  Ensigns  are  getting  a  smaller  share  of  an  already  limited 
resource. 
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4.  Training  Initiative’s  Goals 

The  Division  Officer  at  Sea  Training  Program  Ship’s  Handbook  defines  the 
program  goals.  They  are: 

(1)  To  accelerate  and  enhance  Surface  Warfare  Division  Officer  training  so  that 
the  navy  will  produce  better  prepared  leaders  in  less  time  (p.  1  -3). 

(2)  To  retain  naval  officers  as  career  professionals  (p.  1  -3). 

These  two  statements  are  vague  and  read  more  like  slogans  than 
quantifiable  goals.  For  example,  how  will  accelerated  or  enhanced  training 
produce  better  leaders?  Similarly,  is  it  the  increased  retention  of  one  or  one 
hundred  junior  officers  that  will  fulfill  the  second  program  goal?  It  appears  there 
are  no  established  rubrics,  or  standards,  which  have  been  provided  to 
commanders  to  assist  them  in  determining  whether  the  program  goals  are  being 
met  at  the  shipboard,  squadron,  and  fleet  levels. 

Optimally,  educational  objectives  are  derived  from  the  program’s  goals. 
However,  in  this  instance,  the  learning  objectives  are  taken  from  the  appropriate 
Personnel  Qualifications  Standards  (PQS)  books.  A  PQS  book  identifies  the 
knowledge  and  skills  that  each  officer  must  possess  in  order  to  stand  the 
shipboard  watch  that  the  book  was  written  for.  Consequently,  the  program’s 
goals  have  no  bearing  on  this  training  initiative. 

D.  SURVEY  DATA 

Survey  data  reveals  several  strengths  and  weaknesses  currently  resident 
within  the  Surface  Warfare  School  Command’s  Division  Officer  at  Sea  training 
initiative.  Based  upon  survey  data,  program  strengths  include  Strong 
Educational  Objectives  and  a  strong  Core  Curriculum.  The  Evaluation 
Framework  surveys  identified  three  significant  weaknesses  in  this  training 
initiative.  The  Instructional  Evaluation  category’s  Assessment  function,  the 
Program  Operation  function,  and  the  Instructional  Presentation  function  have 
significant  weaknesses. 
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1.  Program  Strengths 

Based  upon  survey  responses  and  post  survey  interviews,  the  course’s 
primary  strengths  are  its  educational  objectives  and  core  curriculum  functions. 

a.  Educational  Objectives 

Despite  having  ill-defined  program  goals,  the  Educational 
Objectives  are  one  of  the  strengths  of  this  initiative.  This  is  to  be  expected  since 
designers  of  this  CBT  course  simply  transferred  the  course  content  that  used  to 
be  taught  in  the  classroom  to  CD-ROM.  Because  the  traditionally  taught 
curriculum  had  strong  educational  objectives  and  the  information  was  relevant, 
current,  and  useful,  it  stands  to  reason  that  the  CBT  course  would  also  possess 
similar  characteristics.  93.8  percent  of  students  agree  that  learning  objectives 
are  stated  at  the  beginning  of  each  lesson.  Over  half  of  the  students  confirm  that 
the  learning  objectives  are  covered  by  instructional  material.  It  can  be 
concluded,  therefore,  that  the  courseware  is  teaching  the  information  the  Navy 
has  deemed  important  since  lessons  are  covering  the  learning  objectives,  which 
are  derived  from  PQS. 

Despite  the  Educational  Objectives  function  strengths,  the  topics’ 
learning  objectives  can  be  stated  better.  Nearly  half  of  the  students  identified 
words  or  phrases  in  the  learning  objectives  such  as  understand,  list,  explain,  give 
an  example  of,  describe,  identify,  recall,  master,  or  differentiate.  These  words  or 
phrases  identify  objectives  in  terms  of  what  the  teacher  wants  the  student  to  do. 
They  do  not  describe  student  performance  in  terms  of  what  the  student  must  do 
to  satisfy  an  objective.  When  a  student  references  the  learning  objectives  as  a 
checklist  to  verify  that  he  or  she  knows  all  of  the  important  material,  it  makes  it 
easier  on  the  student  if  these  objectives  are  written  from  the  student’s 
perspective. 

b.  Core  Curriculum 

The  Core  Curriculum  is  this  program’s  major  strength.  What  is 
taught  in  this  course  is  relevant,  current,  and  useful.  As  previously  mentioned, 
course  content  was  transferred  to  a  digital  format.  Information  was  neither  to  be 
added  nor  deleted  during  this  transformation  process.  In  its  traditional  format, 
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this  curriculum  was  successfully  taught  to  over  5,000  students  from  1992  to 
2002.  It  is  the  information  in  this  curriculum  that  forms  the  foundation  upon  which 
an  officer’s  professional  knowledge  is  laid. 

Over  80  percent  of  surveyed  students  believe  the  presentations  are 
relevant  to  the  associated  topic  and  will  help  them  perform  their  jobs  better. 
Student  comments  regarding  the  instructional  information  is  very  positive. 
Multiple  students  stated  that  they  believed  the  information  is  very  pertinent, 
beneficial,  and  informative.  Students  also  state  they  believe  the  information  will 
positively  reinforce  what  is  learned  through  OJT. 

75  percent  of  surveyed  Training  Officers  felt  the  material  is  current 
and  relevant  to  today’s  work  environment.  Additionally,  the  majority  of  the 
Training  Officers  believe  the  instructional  material  will  help  students  perform  their 
jobs  better.  In  fact,  one  Training  Officer  reported  that  the  drawings  and  diagrams 
included  in  this  course  are  so  good  that  they  are  now  being  used  for  ship-wide 
training. 

2.  Primary  Weaknesses 

Based  upon  survey  responses  and  post  survey  interviews,  this  program’s 
primary  weaknesses  are  the  Instructional  Evaluation  section’s  Assessment 
function,  the  instructional  presentation  function,  and  the  program  operation 
functions. 

a.  Assessment  Function 

The  Assessment  block  is  rated  poorly  because  ships  are 
experiencing  tremendous  difficulties  with  the  courseware’s  testing  module. 
Testing  is  a  significant  component  to  any  training  program.  It  is  vital  that  these 
technical  difficulties  be  investigated  and  corrected  as  soon  as  possible.  Since  I 
was  unable  to  survey  a  student  or  supervisor  that  had  successfully  taken  or  given 
a  module  test,  I  am  unable  to  adequately  assess  the  function  and  rate  it 
accordingly.  Consequently,  no  further  comments  will  be  made  concerning 
detailed  analysis  in  this  area. 
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b.  Program  Operation  Function 

The  program  operation  function  is  rated  low  for  numerous  reasons. 
First,  during  follow-up  interviews  students  and  supervisors  unanimously  stated 
that  the  software  is  difficult  to  install  and  set  up  for  use.  Second,  ships  are 
experiencing  compatibility  issues  with  the  software.  Third,  security  policies  may 
leave  personal  information  at  risk  for  compromise.  Fourth,  improvements  can  be 
made  to  the  software’s  graphical  user  interface  (GUI). 

1.  Installation  Difficulties.  The  installation  and  set-up 
portions  of  this  CBT  software  need  to  be  automated.  100  percent  of  training 
officers  believed  that  this  CBT  courseware  is  difficult  to  install  and  use.  The 
majority  of  surveyed  students  believed  the  software  was  difficult  to  install  or 
selected  Not  Applicable,  since  they  did  not  install  the  software  or  had  someone 
else  install  it  for  them.  Installation  is  not  intuitive,  even  to  more  experienced 
computer  users.  Some  users,  like  myself,  received  three  CD’s  containing  the 
entire  course.  Others  received  up  to  six  CD’s.  A  few  people  were  able  to 
download  the  courseware  from  a  website.  Installation  currently  involves 
unzipping  files  into  various  folders  and  then  searching  through  these  folders  to 
find  a  set  of  complicated  instructions  for  adding  students  and  using  the  software. 
Zipping  and  unzipping  are  terms  that  describe  the  compression  and 
decompression  of  data.  By  zipping  files,  more  files  can  be  stored  in  a  single 
location.  However,  zipping  a  file  renders  it  unusable  until  it  is  unzipped. 

All  commercial  software  with  multiple  CD’s  includes  an 
automated  installation  process.  When  a  person  inserts  the  first  CD  into  the 
computer,  they  will  have  to  answer  a  short  series  of  questions  and  then  the 
software  automatically  installs  itself.  The  users  have  only  to  then  remove  and 
insert  requested  discs  as  the  program  is  installed. 

2.  Software  Compatibility.  Ships  are  also  reporting  minor 
compatibility  issues.  The  SWOS  CBT  courseware  requires  the  use  of  Microsoft 
2000  series  of  products  to  function  properly.  Many  of  the  ships  I  visited  did  not 
have  the  required  LAN  infrastructure  to  support  the  courseware’s  Microsoft  2000 
series  requirement.  Due  to  increasingly  tight  budget  constraints  and  licensing 
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issues,  it  seems  doubtful  that  these  ships  will  start  utilizing  the  required  Microsoft 
products  in  the  foreseeable  future. 

3.  Security  Policies.  Training  Officers  report  that  all  student 
data  is  stored  on  a  computer,  or  computers,  that  have  direct  access  to  the 
Internet.  Even  if  data  is  stored  on  a  computer  that  resides  behind  a  firewall,  it  is 
well  known  that  the  data  is  not  secure  if  that  computer  has  direct  access  to  the 
Internet.  There  are  two  possible  solutions  for  quickly  and  inexpensively 
improving  security  and  lowering  the  risk  of  compromising  sensitive  information. 
First,  move  the  data  to  a  computer  that  cannot  be  connected  to  the  Internet.  This 
is  known  as  air-gapping.  Transmit  and  receive  the  data  via  computers  that  are 
connected  to  the  Internet,  but  move  it  by  floppy  diskette  to  an  air-gapped 
machine  for  storage.  This  is  what  most,  if  not  all,  major  banks  have  resorted  to  in 
order  to  decrease  the  risk  of  compromise.  The  second  option,  and  the  one  I 
personally  favor,  is  to  encrypt  all  student  data  and  leave  it  encrypted  while  it  is 
stored  on  the  computer’s  hard  drive.  There  are  very  strong  encryption  algorithms 
available  that  cannot  easily  be  broken  using  even  advanced  techniques  that 
would  serve  this  purpose  very  well. 

4.  Graphical  User  Interface.  During  post  survey  interviews, 
the  students  made  two  recommendations  for  improving  the  courseware’s  GUI. 
Although  mixed  results  appear  on  the  survey  (40  percent  said  they  could  not 
track  lesson  progress),  students  desire  a  graphical  representation  of  their 
progress  through  each  lesson.  This  can  be  accomplished  with  a  graphical  dot  on 
a  sliding  bar  across  the  top  of  the  computer  screen.  It  will  help  the  students  to 
judge  how  far  they  have  come  in  each  lesson  and  my  serve  as  a  positive 
motivator  to  continue  through  the  end  of  each  lesson.  The  second 
recommendation  is  to  link  all  the  related  lessons  in  such  a  way  as  to  make  them 
accessible  from  the  end  of  all  of  the  lessons  that  are  related.  Many  times,  a 
group  of  three  or  four  topics  will  be  completed  by  the  students  and  they  have  to 
exit  out  of  the  GUI  to  select  another  related  topic  in  the  same  module.  The 
students  stated  this  was  a  source  of  irritation  that  served  to  demotivate  them  over 
time. 
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c.  Instructional  Presentation 

The  instructional  presentation  function  contains  eight  elements. 
Four  of  these  elements,  Text,  Questioning,  Lesson  Sequencing,  and  Color, 
received  favorable  marks  by  the  majority  of  those  surveyed.  Unfortunately,  three 
of  the  four  remaining  elements,  Interactivity,  Environment,  and  Instructional 
Quality,  received  poor  marks  by  those  taking  the  survey. 

1.  Interactivity.  A  highly  interactive  lesson  is  one  that  will 
engage  the  students,  raise  their  curiosity  and  interest,  and  positively  motivate 
them  to  learn.  It  does  this  by  forcing  students  to  act  upon  new  information 
multiple  times,  significantly  increasing  the  probability  that  the  information 
becomes  resident  in  long-term  memory  and  semantic  maps.  76  percent  of 
students  disagreed  with  a  survey  item  asking  if  lessons  kept  their  attention. 
Additionally,  the  same  percentage  of  students  disagreed  with  a  survey  item 
asking  if  the  lessons  were  entertaining.  11  of  the  17  students  reported  that  they 
were  frequently  bored  while  using  the  instructional  software.  The  students 
complain  that  there  is  too  much  reading  that  has  to  be  done  on  screen. 

Students  who  are  bored  and  uninterested  are  more  likely 
than  motivated  students  to  page  through  lessons  as  fast  as  they  can  and  devise 
other  methods  for  learning  the  material,  or  worse,  figure  out  how  to  survive 
without  learning  the  material.  Because  of  this,  it  is  important  that  the 
presentations  be  improved,  making  them  more  interactive  and  entertaining  to  the 
students.  A  previous  chapter  details  the  benefits  of  multi-modal  presentations. 
The  increased  use  of  audio  narration,  movies,  video  clips,  and  more  multi¬ 
layered  interactive  pictures  in  conjunction  with  the  textual  content  is  a  good  place 
to  start  improving  the  lessons. 

2.  Environment.  This  element  receives  a  low  grade  for  two 
reasons.  Students  are  having  a  difficult  time  finding  instructional  resources  that 
are  available  long  enough  to  complete  most  lessons.  Additionally,  students  are 
often  interrupted  or  called  away  in  the  middle  of  lessons . 

Every  student  I  surveyed  told  me  that  resources  were 
scarce.  70  percent  of  surveyed  students  said  that  they  did  not  have  immediate 
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access  to  instructional  resources  such  as  computers,  software,  books, 
instructions,  or  manuals.  As  previously  mentioned,  computers  are  in  high 
demand  and  short  supply.  Ship’s  business  takes  priority  over  Ensign  training 
time.  According  to  students,  ships  have  not  received  any  more  manuals,  despite 
now  being  burdened  with  training  newly  commissioned  officers.  While  underway, 
manuals  that  are  needed  to  study  are  also  needed  by  the  watch  standers  in 
performance  of  their  duties.  Consequently,  students  do  not  get  to  study  many  of 
the  manuals  while  a  ship  is  underway.  In  the  traditional  classroom  curriculum, 
each  student  was  provided  a  safe  containing  a  copy  of  every  manual  that  was 
needed  to  learn  from. 

Only  two  of  surveyed  students  have  been  able  to  start  and 
complete  lessons  without  interruption.  Over  half  of  the  students  report  that  they 
are  frequently  called  away  in  the  middle  of  lessons.  It  is  difficult  for  students  to 
take  in  new  information,  move  that  information  through  their  STSS  into  working 
memory,  and  act  on  it  long  enough  to  store  it  in  long-term  memory,  if  they  are 
consistently  being  interrupted  or  called  away  in  the  middle  of  their  studies. 

65  percent  of  students  and  75  percent  of  supervisors  feel 
that  a  different  location  can  or  should  be  used  for  learning  with  this  CBT 
software.  The  students  and  their  training  officers  contend,  and  I  agree,  that  there 
should  be  a  centralized  location,  or  learning  resource  room,  on  the  waterfront 
where  they  can  go  to  study  while  their  ship  is  in  its  homeport.  This  would  allow 
students  learn  in  a  quiet,  distraction  free  environment  while  taking  the  CBT 
lessons.  When  finished,  the  students  can  go  back  to  their  ships  and  receive 
reinforcing  training  via  practicums  or  walk-throughs. 

3.  Instructional  Quality/Effectiveness.  Comparing  the 
strengths  and  weaknesses  within  the  Pedagogical  Evaluation  and  Instructional 
Evaluation  sections  determines  the  instructional  Quality  and  effectiveness  of  the 
CBT  initiative.  The  following  paragraphs  summarize  the  previously  discussed 
key  points  that  are  relevant  to  this  function’s  rating. 

The  strength  of  this  initiative  rests  squarely  in  its  core 
curriculum  function.  The  information  that  is  being  taught  to  the  students  is 
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relevant,  current,  and  useful.  However,  this  may  not  be  enough  to  ensure  the 
success  of  this  program.  Initial  difficulties  that  Ensigns  are  experiencing  are  not 
making  a  favorable  impression  of  the  warfare  community  on  them.  These 
Ensigns’  difficulties  and  initial  impressions  of  this  training  initiative  may  ultimately 
lead  to  decreased  retention  within  the  community. 

The  lessons  lack  needed  elements  of  excitement  and 
interactivity.  The  Ensigns  report  that  they  are  often  bored.  The  students  have 
many  duties,  often  sacrificing  needed  sleep  to  take  a  CBT  lesson,  and  are 
frequently  disappointed  by  the  way  the  information  is  delivered.  Additionally, 
students  are  having  a  difficult  time  finding  available  computers  to  use,  and  when 
they  are  able  to  use  one,  are  frequently  interrupted  or  called  away. 

Students  must  often  rely  on  department  heads,  peers,  and 
watch  standers  for  their  instruction.  These  people  are  all  very  busy,  and  do  not 
necessarily  have  the  time  required  to  properly  teach  a  concept  or  skill. 
Additionally,  these  sailors  may  not  have  any  instructor  training  or  have  the 
requisite  ability  to  effectively  teach  or  train  others.  For  example,  the  junior 
Quartermaster  standing  Quartermaster  of  the  Watch  on  the  ship’s  bridge,  may 
not  have  the  time  or  the  ability  to  thoroughly  explain  and  answer  questions  about 
set,  drift,  dead  reckoning,  and  aspects  of  celestial  navigation  to  an  Ensign  the 
comes  to  the  bridge  for  training. 

The  students  also  One  of  the  most  telling  data  points  is  that 
all  of  the  Training  Officers  believe  that  the  program  does  not  reduce  the  time  it 
takes  to  learn  Surface  Warfare  Officer  skills.  If  learning  time  is  not  reduced  how 
can  the  first  program  goal  ever  be  achieved?  As  a  result  of  the  above 
discussion,  Instructional  Quality  and  Effectiveness  is  rated  as  very  low. 
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V.  RECOMMENDATIONS 


A.  FURTHER  RESEARCH 

IDSI  delivered  this  CBT  initiative  to  the  Navy  early  this  year.  The  majority 
of  new  accessions  into  the  Surface  Warfare  community  are  May  and  June 
graduates.  Consequently,  most  new  students  are  just  starting  their  course  of 
instruction  with  this  CBT  software.  To  date,  nobody  has  fulfilled  all  of  the 
requirements  of  this  program  and  earned  their  Surface  Warfare  Pin.  It  is 
recommended  that  this  program  be  tracked  and  compared  to  the  traditional 
classroom  course  of  instruction  in  two  broad  areas. 

1.  Knowledge  Value  Added  Analysis 

First,  conduct  a  Knowledge  Value  Added  analysis  on  the  two  curriculums 
to  verify  that  this  CBT  initiative  is  generating  an  appropriate  return  on  investment. 
If  not,  this  process  will  allow  evaluators  to  identify  which  processes  in  the 
program  need  to  be  modified  or  eliminated  in  order  to  then  generate  appropriate 
returns  on  the  surface  warfare  community’s  investment.  Since  this  program  is  in 
its  infancy,  and  has  no  graduates,  there  is  currently  not  enough  available  data  to 
support  an  analysis  of  this  kind  at  present. 

2.  Track  and  Compare  Student  Progress  with  Historical  Data 

Second,  track  the  students’  progress  from  the  time  they  arrive  at  their  first 

ships  to  the  time  they  earn  their  CIC  Watch  Officer  and  Officer  of  the  Deck 
(underway)  qualifications  and  also  their  warfare  pins.  Compare  these  results  to 
the  historical  qualification  data  with  students  that  graduated  from  SWOS  in 
Newport,  Rhode  Island. 

B.  NON-SOFTWARE  RELATED  RECOMMENDATIONS 

Following  a  review  of  personal  observations  and  an  analysis  of  survey  and 
interview  data,  three  recommendations  are  provided.  First,  it  is  recommended 
that  the  primary  surface  warfare  training  activity  on  each  coast  be  directed  to 
conduct  thorough  evaluations  of  ships’  implementation  and  execution  progress 
as  long  as  Ensigns  are  trained  via  this  program.  For  example,  it  is  recommended 
that  the  Atlantic  Training  Group  (ATG)  be  designated  as  the  East  Coast’s  training 
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command  responsible  for  evaluating  ships’  implementation  and  execution  of  this 
CBT  initiative.  Second,  It  is  recommended  that  a  waterfront  study  space/learning 
resource  center  (LRC)  be  established  in  every  fleet  concentration  area  so  that  a 
distraction  free,  off-ship  learning  environment  is  created  for  all  students  in  this 
program.  Third,  it  is  recommended  that  increased  resources,  including 
computers,  Navy  instructions,  publications,  and  technical  manuals  be  provided  to 
all  ships  for  the  exclusive  use  of  junior  officers  enrolled  in  this  program. 

1.  Include  Training  Commands 

Involving  the  primary  training  commands  on  each  coast  accomplishes 
several  things  vital  to  the  success  of  this  program.  First,  it  places  much  needed 
help  and  assistance  near  the  waterfront  on  both  coasts.  The  training  officers 
complained  that  they  have  been  unsuccessful  getting  the  help  they  need  to  fully 
implement  this  program.  Personnel  at  the  training  command  would  be  trained 
how  to  properly  install  and  run  the  software,  and  would  render  assistance  to 
nearby  ships  when  it  is  needed.  A  Lieutenant  Commander,  with  the  assistance 
of  two  or  three  Lieutenants,  and  a  handful  of  skilled  enlisted  specialists  at  each  of 
these  training  commands  would  be  able  to  perform  the  requisite  tasks  associated 
with  this  CBT  initiative. 

Second,  by  empowering  the  training  commands  to  conduct  annual  or 
semi-annual  program  evaluations,  it  ensures  that  all  Ensigns  receive  the 
necessary  and  quality  instruction  that  is  needed  to  transform  them  into  the 
outstanding  professional  mariners  that  the  Navy  is  known  for. 

Third,  it  allows  waterfront  seminars  to  be  conducted  on  a  routine  basis 
with  little  expense  to  the  government.  These  seminars  would  allow  supervisors 
and  students  to  act  collaborativeiy  to  solve  problems,  increase  awareness, 
discuss  cross-decking  opportunities,  and  improve  the  quality  of  junior  officers’ 
training  regimens. 

2.  Create  Waterfront  Learning  Resource  Centers 

The  creation  of  a  waterfront  learning  resource  center  is  very  beneficial  for 

students  in  this  program.  At  present,  students  are  finding  it  very  difficult  to 

complete  lessons  without  interruption.  A  quiet,  distraction  free  environment  will 
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improve  retention,  and  subsequently,  learning  transfer,  among  the  students. 
Additionally,  instructional  resources  would  be  in  a  centralized,  easily 
maintainable  location.  In  times  of  fiscal  constraint,  a  twenty-computer  LAN  that  is 
located  in  an  LRC  is  less  costly  than  two  or  three  computers  placed  on  every 
ship,  and  is  much  more  economical  than  buying  a  laptop  for  every  student. 
Finally,  a  LRC  allows  students  to  work  together  and  learn  coliaboratively,  which 
research  has  shown  will  lead  to  improved  retention  and  performance. 

3.  Increase  Instructional  Resources 

It  is  important  that  students  have  nearly  ubiquitous  access  to  computers 
while  they  are  enrolled  in  a  mandatory  self-paced  Computer-based  training 
program.  There  are  not  enough  computers,  publications,  and  other  instructional 
material  on-hand  to  ensure  that  the  students  receive  the  same  quality  of  training 
as  was  received  when  it  was  conducted  in  Newport,  Rhode  Island.  Not  supplying 
the  students  with  the  resources  necessary  for  successful  completion  of  the 
program  is  akin  to  shooting  yourself  in  the  foot.  Since  these  officers  are  the 
future  of  this  warfare  community,  it  is  important  that  the  Navy  supply  them  with 
the  necessary  tools  to  complete  this  course  of  study. 

C.  SWOS  DOC  CBT  SOFTWARE  RELATED  RECOMMENDATIONS 

Three  recommendations  related  to  the  software  are  provided.  First,  and 
most  important,  it  is  recommended  that  the  courseware  presentations  be 
modified  and  made  more  interactive  and  exciting.  Second,  it  is  recommended 
that  the  software  be  modified  to  encrypt  all  data  prior  to  saving  it  on  either  a 
portable  media  or  fixed  hard  drive. 

1.  Improve  Instructional  Presentations 

It  is  important  that  the  lesson  presentations  be  improved.  Students 
unanimously  reported  that  lessons  are,  despite  containing  useful  information, 
very  boring.  More  interactive  activities  need  to  be  included  in  the  lessons. 
These  include,  but  are  not  limited  to  numerous  questioning  techniques,  point, 
click  and  drag  activities,  and  video  segments  that  offer  students  a  choice  at  their 
end.  Each  choice  starts  another  video  segment,  so  the  student  can  see  and  hear 
the  consequences  of  the  choices  made.  To  make  the  lessons  less  dull,  more 
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video,  animation,  interactive  pictures,  and  audio  narration  should  be 
incorporated.  Many  successful  CBT  training  packages  include  audio  narration. 
Additionally,  audio  narration  makes  presentations  bimodal,  and  as  mentioned  in 
a  previous  chapter,  improves  student  retention.  By  improving  the  lessons’  level 
of  interaction  and  making  them  exciting,  student  curiosity,  interest,  and 
motivation  will  be  improved.  As  a  result,  more  information  will  be  acted  upon, 
memory  function  will  increase,  and  more  transfer  will  occur.  The  end  result  will 
be  better-trained  officers  that  are  able  to  apply  more  of  what  they  learned  via  the 
CBT  software  to  very  complex  real-life  situations. 

2.  Add  an  Encryption  Standard 

By  adding  an  automated  encryption  scheme  to  the  CBT  software  package, 
the  Navy  will  significantly  increase  the  safety  of  personal  and  performance 
information  stored  on  its  computers.  Students’  personal  and  performance  data 
will  be  collected,  transmitted,  and  warehoused  multiple  times  throughout  the 
duration  of  this  training  initiative.  By  keeping  the  data  encrypted,  and  decrypting 
it  only  when  reading  or  performing  maintenance  on  the  files,  hackers  and  other 
individuals  with  malicious  intent  will  be  unable  to  decipher,  read,  alter,  or  use  any 
data  that  may  be  intercepted  or  otherwise  compromised. 

D.  CONCLUSION 

By  implementing  the  recommendations  that  are  presented  here,  the  Navy 
will  transform  this  CBT  training  initiative  from  a  lackluster  performer  into  a  shining 
example  of  what  an  organization  can  accomplish  through  the  use  of  Computer- 
based  training  that  is  reinforced  through  real  world  experience  while  on  the  job. 
Implementation  of  the  following  recommendations  will  achieve  the  following: 

(1)  Student  interest  and  motivation  will  improve.  Learning  transfer  will  improve, 
and  consequently,  performance  will  improve. 

(2)  The  programs  objectives  will  be  met  because 

(a)  Qualification  times  for  all  watch  stations  will  be  reduced.  As  a  result,  the 
program’s  first  stated  goal  of  better-prepared  Surface  Warfare  Officers  in  less 
time  will  become  a  reality. 
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(b)  An  increase  in  qualified  officers  amounts  to  more  watch  sections  while 
underway.  As  a  result  of  the  increase  in  watch  sections,  each  officer  has  more 
time  off  to  pursue  other  qualifications  and  interests.  As  a  consequence,  morale 
among  the  junior  officers  will  increase,  leading  to  better  retention.  This  satisfies 
the  program’s  second  goal  of  improving  retention  rates  among  in  its  officer  corps. 
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APPENDIX  A.  STUDENT  AND  SUPERVISOR  SURVEYS 


A.  STUDENT  SURVEY 

STUD  ENT’S  SURVEY  OF  INSTRUCT  ION 

Please  mark  you  r  choices  by  pi  acing  a  checkmark  in  the  app  ropriate  box  for  each  of  the 
following  items.  You r  specific  answers  will  remain  completely  anony  mous ,  but  your 
views,  in  combination  with  those  of  others,  are  extremely  important. 

SECTION  1:  YOUR  BACKGROUND  AND  USEOF  COMPUTERS 

1.  What  is  your  ag  e? 

Unde  r  25 . •  26  to  30 . •  31  to  35 . •  36  to  40 . • 

41  or  older . • 

2.  What  is  your  Gender? 

Male. . •  Female . • 

3.  What  is  your  primary  position  with  respect  to  t  his  training  initiative? 

Student . •  Instructor . •  Administrator  /  Supervisor . • 

4.  How  would  you  rate  your  computer  skills? 

Novice  or  Beginn  er . •  Intermediate . • 

Advanc  ed  or  Expert . • 


SEC TION  II:  INSTRUCT!  ONAL  QUALITY  ASSESSMENT 


Please  circle  the  number  that  best  represents  your  view  regarding  the  content  of  the 
SWOS  DIVISION  OFFICER  COURSEWARE  . 

If  a  particular  item  is  Not  Applicable,  please  circle  NA . 

If  you  Strongly  agree  with  the  statement,  please  circle  the  Five  (5). 

If  you  agree  w  ith  a  statement,  please  circle  Four  (4). 

If  you  neither  agree  or  disagree  with  an  item,  please  circle  Three  (3). 

If  you  Disagree  with  a  statement,  please  circle  Two  (2). 

If  you  Strongly  Disagree  with  a  statement,  please  Circle  One  (1). 


1 .  I  unde  rstood  the  pu  rpose  of  the  cou  rse 

2.  The  learning  objectives  were  clearly  stated 

3.  The  information  that  was  presented  was  relevant  to  the  topic 


NA 

NA 

NA 


1....2....3....4....5 


....1.. 

..2.. 

..3.. 

.A.... 

....1.. 

..2.. 

..3.. 

.A.... 
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4.  Learning  objectives  were  li  sted  at  the  beginning  of 
every  lesson 

5.  All  of  the  learning  objectives  were  covered  by  the 
instructional  material 

6.  Learning  objectives  include  key  words  or  phrases  such  as 
create,  install,  set-up,  start,  repair,  diagno  se,  troubleshoo  t, 
organize,  or  write 

7.  Learning  objectives  include  key  words  or  phrases  such  as 
unde  rstand ,  list ,  exp  lain,  give  an  example  of,  describe, 
identif  y,  recall,  master,  or  differentiate 

8.  All  of  the  information  that  was  required  to  satis fy  stated 
performance  criteria  was  provided  in  the  lessons 

9.  The  informa  tion  I  learned  will  help  me  perform  better 

10.  I  found  the  material  in  this  course  very  useful 

11.  Test  questions  were  based  upon  the  learning  objectives 

12.  Test  questions  only  covered  items  I  had  been  laugh  t 

13.  Directions  for  each  test  were  clear 

14. 1  unde  rstood  the  performance  standard  for  each  test  or  quiz 

15.1  unde  rstood  the  performance  standard  for 
each  graded  activity 

16.  Tests  were  fairly  and  properly proctored  /  administered 

17. 1  received  timely  feedback  about  my  test  scores 

18.  The  feedback  I  received  regarding  tests  was  valuable 

19.  The  feedback  I  received  regarding  my  test  scores 
will  help  me  improve  my  performance 

20.  Graded  activities  were  fairly  and  properly  administered 

21.1  received  timely  feedback  regarding  my  graded  activities 

22.  The  feedback  I  received  regarding  my  graded  activities 
was  valuable 

23.  The  feedback  I  received  regarding  graded  activities  will 
help  me  improve  my  performance 

24.  The  software  wa  s  easy  to  install 

25.  The  software  wa  s  easy  to  use 

26.  The  software  menu  was  easy  to  find 


NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 
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27.  The  software  wa  s  easy  to  navigate 

28.  Directions  were  easy  to  find 

29.  Directions  were  easy  to  follow 

30.  On-screen  options  were  easily  und  erstood 

31.  The  software  had  an  integrated  search  function 

32. 1  always  knew  where  to  look  for  on-screen  help 
33.  Human  help  was  always  available 

34. 1  was  always  ab  le  to  page  forward 

35. 1  was  always  ab  le  to  page  back 

36. 1  was  able  to  bookm  ark  my  location  and  come  back  later 

37. 1  was  able  to  track  my  progress  though  each  lesson 

38. 1  was  able  to  track  my  progress  through  the  training  program 

39. 1  was  able  to  easily  generate  practice  tests 
40.  Each  lesson  kept  my  attention 

41.1  though  t  the  lessons  we  re  entertaining 

42.  The  multim  edia  used  in  each  lesson  was  r  elevant 

43.  Lessons  incorporated  audio  narration 

44.  High  -re  solution  graphics  /  pictures  were  incorporated 
into  each  lesson 

45.  The  use  of  relevant  video,  animation,  andp  retires  were 
incorporated  into  each  lesson 

46.  The  use  of  video,  animation,  and  pictures  in  eachl  esson 
kept  my  interest 

47.  There  were  too  many  videos,  animations  or  pictures 
in  each  lesson 

48.  The  information  ineach  lesson  wa  s  worthwhile 

49.  The  cou  rseware  taugh  t  me  useful  information 

50.  After  using  the  courseware,  I  understand  the  material  better 

51. 1  was  frequently  bored  whileusing  the  instructional 
software 

52.  The  use  of  color  enhanc  ed  the  instructional  presentations 

53.  Contrasts  in  the  use  of  co  lor  made  text  easier  to  read 


NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 
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54. 1  always  had  access  to  a  quiet,  comfortable, 
distraction  free  learning env  ironment 

NA....1....2....3....4....5 

55. 1  was  frequently  interrupted  or  called  away  while  using 
tire  cour  seware 

NA....1....2....3....4....5 

56. 1  was  able  to  start  and  complete  a  lesson  without 
interrup  tion 

NA....1....2....3....4....5 

57. 1  always  had  immediate  access  to  instmctional  resources 
such  as  computers,  software,  books,  instructions, 
or  manua  Is 

NA....1....2....3....4....5 

58.  A  different  location  can  or  should  be  used  fo  r  learning 
via  this  CBT  software 

NA....1....2....3....4....5 

59. 1  was  able  to  easily  read  textual  information 

NA....1....2....3....4....5 

60. 1  had  to  scroll  down  through  the  text  that  was  on  the  screen 

NA....1....2....3....4....5 

61 .  The  que  stions  I  was  asked  focused  on  information  I  had 
already  been  taught 

NA....1....2....3....4....5 

62.  The  que  stions  I  was  asked  focused  on  information  that 

I  was  about  to  learn 

NA....1....2....3....4....5 

63.  The  que  stions  I  was  asked  were  related  to  the  topic’s 
learning  objectives 

NA....1....2....3....4....5 

SECTION  HI:  SHORT  ANSWER 


Please  legjbly  print  your  answers  to  the  following  questions.  There  are  no  correct  or 
incorrect  re  spouses.  However,  specific  and  detailed  answers  are  appreciated.  If  a 
question  is  not  applicable  to  your  situation,  either  leave  the  answer  blank  or  use 
N/A.  Your  specific  answers  will  remain  completely  anonymous. 


1.  Have  you  ever  failed  to  meet  the  required  standards  for  an  activity  or  a  test?  If  yes, 
why  do  you  believe  that  you  failed  to  meet  the  required  standards? 


2.  If  you  an  swered  yes  to  the  above  question,  did  you  receive  remediation?  If  yes,  why 
was  the  remediation  you  received  either  effective  or  inelf ective? 
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3.  Wha  t  are  the  best  parts  of  this  cour  se  and  why  ? 


4.  What  are  the  worst  parts  of  this  course  and  why? 
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5.  What  should  be  done  to  improve  this  course  of  instruction? 


THANK  YOU  FOR  TAKING  THE  TIME  TO  COMPLETE  THIS  SURVEY. 
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B.  SUPERVISOR  SURVEY 


SUPERVISOR’S  SURVE  Y  OF  INSTRUCTION 


Please  mark  you  r  choices  by  placing  a  checkmark  in  the  appropriate  box  for  each  of  the 
folb wing  items.  Your  specific  answers  will  remain  completely  anony  mous ,  but  your 
views,  in  combination  with  those  of  others,  are  extremely  important. 


SECTION  1:  YOUR  BACKGROUND  AND  USE  OF  COMPUTERS 

1.  What  is  your  age? 

Under  25 . •  26  to  30 . •  31  to  35 . •  36  to  40. . 

41  to  45  . •  45  to  50 . •  Over  50 . • 

2.  What  is  your  Gender? 

Male . •  Female . • 

3.  What  is  your  primary  position  with  respect  to  this  training  initiative? 

Student . •  Instructor . •  Administrator  /  Supervisor . 

4.  How  would  you  rate  your  computer  skills? 

Novice  or  Beginn  er . •  Intermediate . • 

Advanc  ed  or  Expert . • 


SECTION  II:  INSTRUCTIONAL  QUALITY  ASSESSMENT 


Please  circle  the  number  that  best  represoits  your  view  regarding  the 
SWOS  DIVISION  OFFICER  COURSEWARE. 

If  a  particular  item  is  Not  Applicable,  pleasecircle  NA. 

If  you  Strongly  agree  with  the  statement,  please  circle  the  Five  (5). 

If  you  agree  with  a  statement,  please  circle  Four  (4). 

If  you  neither  agree  or  disagree  with  an  item,  pleasecircle  Three  (3). 

If  you  Disagree  with  a  statement,  please  circle  Two  (2). 

If  you  Strongly  Disagree  with  a  statement,  please  Circle  One  (1). 


1.  The  SWOSCOLCOM  Division  Officer  At  Sea  Training 

Initiative’s  goals  are  well  defined  NA....  1....2....3....4....5 

2.  I  know  where  the  SWOSCOLCOM  Division  Officer  At  Sea 

Training  Initiative’s  goals  are  defined  NA....  1....2....3....4....5 
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3.  The  Training  Initiative ’s  goals  do  not  confli  ct  with  other 
organ  iza  ti  ona  1  go  al  s 

4.  This  course  reduces  the  students’  OJT  requirements 

5.  This  course  reduces  the  time  it  takes  students  to  learn 
Surface  Warfare  Officer  job  skills 

6.  Each  topic  has  associated  learning  objectives 

7.  Learning  objectives  are  presented  at  the  beginning  of 
every  lesson 

8.  Learning  objectives  include  key  words  or  phrases  such  as 
create,  install,  set-up,  start,  repair,  diagno  se,  troubleshoo  t, 
organize,  or  write 

9.  Learning  objectives  include  key  words  or  phrases  such  as 
understand ,  list ,  exp  lain,  give  an  example  of,  describe, 
identif  y,  recall,  master,  or  differentiate 

10.  The  learning  objectives  support  the  organizational  training 
initiative’ s  goals 

11.  Test  questions  are  derived  from  the  learning  objectives 

12.  Student  performance  standards  are  derived  from  the 
learning  objectives 

13.  All  student  performance  standards  are  published 

14.  Students  unde  rstand  the  grading  criteria  for  each  and 
every  test  and  graded  activity 

15.  Information  taugh  t  to  the  students  is  relevant  to  the  topic 

16.  The  instructional  presentations  cover  all  the  learning 
objectives 

17.  Instructiona  l  material  is  current  and  up  to  date 

18.  Instructional  material  will  help  the  students  perform  their  jobs 
better 

19.  Directions  for  each  test  and  graded  activity  are  clear 

20.  Tests  and  graded  activities  are  fair 

21.  Remediation  is  available  for  students 

22.  Remediation  for  failing  to  meet  a  standard  is  effective 

23.  Each  test  proctor  or  activity  participant  other  than  the 
student  is  identified  in  advance 


NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 
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24.  Each  test  proctor  or  activity  participant  other  than  the 
student  is  trained  and  qualified  for  the  part  of  the  test 
or  activity  they  are  participating  in 

25.  AH  tests  are  given  and  graded  by  the  computer 

26.  For  all  test  items  not  graded  by  a  computer,  the 
criteria  for  a  fully  correct  answer  is  known  by  the  grader 

27.  For  all  test  items  not  graded  by  a  computer,  the  performance 
criteria  for  fully  correct  answers  are  establis  hed 

28.  For  all  test  items  not  graded  by  a  computer,  the  criteria  for 
afully  correct  answer  is  used  to  grade  a  particular  test  item 

29.  For  all  test  items  not  graded  by  a  computer,  the  same  person 
that  writes  the  test  also  grades  it 

30.  For  all  performance  based  ac  tivities,  the  criteria  for  correctly 
performed  actions  is  known  by  the  grader 

3F  For  all  performance  based  ac  tivities,  the  criteria  for  correctly 
performed  actions  is  always  strictly  used  by  the  grader 

32.  The  collection  of  student  test  and  ac  tivity  performance 
data  enhanc  es  organ  izational  performance 

33.  Students  receive  timely  feedback  regarding  their  performance 

34.  Feedback  to  students  is  relevant  and  useful  to  them 

35.  Feedback  to  students  either  praises  them  or  identifi  es  a 
weakness  and  reveals  how  to  correct  that  deficiency 

36.  The  software  /  courseware  is  easy  to  install 

37.  The  software  /  courseware  is  easy  to  us  e 

38.  Student  progress  is  easily  tracked  via  the  courseware 

39.  The  courseware  adequately  protects  personal  information 

40.  Files  where  persona  1  or  performance  in  formation  is  stored 
are  password  protected 

4F  Personal  and  performance  information  is  encrypted  during 
trans  fers  between  locations 

42.  Personal  and  performance  data  is  stored,  updated,  and 
maintained  on  a  computer  that  has  direct  access  to  the  Internet 

43.  Personal  and  performance  data  is  only  downloaded  on  a 
computer  that  has  direct  access  to  the  Internet,  but  stored  on 
a  computer  that  has  zero  Internet  access  (air  gapped) 

44.  Students  are  provided  a  quiet,  distraction  free  environment 


NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

NA....1....2....3....4....5 
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45.  Students  are  not  interrupted  or  called  away  while  using  the 


courseware  to  learn 

NA....1....2....3....4....5 

46.  A  different  location  can  or  should  be  used  fo  r  learning 
via  this  computer-based  training  software 

NA....1....2....3....4....5 

47.  This  computer-based  training  program’s  administrators  and 
supervisors  are  all  adequately  trained 

48.  Evaluation  standards  for  assessing  this  computer-based 
training  initi  ative  hav  e  been  estabH  shed 

NA....1....2....3....4....5 

NA....1....2....3....4....5 

49.  Evaluation  standards  for  assessing  this  computer-based 
training  initiative  have  been  promulg  ated 

NA....1....2....3....4....5 

50.  The  standards  for  this  computer-based  training  initiative 
are  being  met  or  exceeded 

NA....1....2....3....4....5 

51 .  Formal  standards  are  established  for  the  presentation  of 
this  training  initiative’s  instructional  material  by  means 
other  than  the  computer  (watch  standers) 

NA....1....2....3....4....5 

52.  Instructional  presentations  of  this  training  initiative’s 
material  by  means  other  than  a  computer  is  closely 
monitored 

NA....1....2....3....4....5 

SECTION  HI:  SHORT  ANSWER 


Please  legibly  print  your  answers  to  the  following  questions.  There  are  no  correct  or 
incorrect  re  spouses.  However,  specific  and  detailed  answers  are  appreciated.  If  a 
question  is  not  applicable  to  your  situation,  either  leave  the  answer  blank  or  use 
N/A.  Your  specific  answers  will  remain  completely  anonymous. 


1.  If  the  SWOSCOLCOM  Division  Officer  At  Sea  Training  Initiative’s  goals  can  be 
better  defined,  how  would  you  change  them  to  make  them  better? 
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2.  How  are  the  students’  performance  standards  for  graded  activities  establis  hed?  Are 
the  se  standards  uni  form  throughou  t  the  fleet? 


3.  What  is  done  with  the  collected  student  test  and  activity  performance  data? 


4.  Doe  s  the  collection  of  student  test  and  activity  performance  data  enhance  or  bene  lit 
organ izationa  1  performance?  If  so,  how? 


5.  Doe  s  the  organization  require  periodic  evaluations  of  this  training  initiative?  If  so, 
wha  t  aspects  of  this  Computer-based  training  initiative  are  to  be  evaluated? 


6.  What  is  the  procedure  for  reporting  we  aknesses  or  deficiencies  that  are  discovered  in 
the  SWOSCOLCOM  Division  Officer  At  Sea  Training  Initiative? 


7.  If  a  weakne  ss  or  deficiency  is  discovered  in  the  SWOSCOLCOM  Division  Officer  At 
Sea  Training  Initiative,  wha  t  is  the  maximum  time  that  is  allotted  to  correct  it? 
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8.  Are  standards  established  for  the  presentation  of  instructional  material  via  CBT?  If 
yes,  please  explain  why  o  r  how  those  standards  have  or  have  no  t  been  met. 


9.  What  is  /  are  the  major  strengths  of  this  CBT  initiative? 


10.  What  is  /  are  the  major  weaknesses  of  this  CBT  initiative? 
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11.  What  should  be  done  to  improve  this  course  of  instruction? 
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THIS  PAGE  INTENTIONALLY  LEFT  BLANK 
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APPENDIX  B.  COMPLETE  SURVEY  DATA 


A.  RESPONSES  TO  STUDENT  SURVEY,  PART  II 


NUMBER 

QUESTION 

N/A 

i 

2 

3 

4 

5 

1 

I  understood  the  purpose  of  this  course 

0 

0 

1 

2 

10 

4 

2 

The  learning  objectives  were  clearly  stated 

0 

0 

2 

3 

9 

3 

3 

The  information  that  was  presented  was  relevant  to 
the  topic 

1 

0 

0 

2 

n 

3 

4 

Learning  objectives  were  listed  at  the  beginning  of 
every  lesson 

1 

0 

0 

0 

10 

5 

5 

All  of  the  learning  objectives  were  covered  by  the 
instructional  material 

1 

0 

1 

5 

6 

3 

6 

Learning  objectives  include  key  words  or  phrases 
such  as  create,  install,  set-up,  start,  repair, 
diagnose,  troubleshoot,  organize  or  write 

2 

3 

3 

6 

2 

0 

7 

Learning  objectives  include  key  words  or  phrases 
such  as  understand,  list,  explain,  give  an  example 
of,  describe,  identify,  recall,  master,  or  differentiate 

2 

1 

1 

5 

5 

2 

8 

All  of  the  information  that  was  required  to  satisfy 
stated  performance  criteria  was  provided  in  the 
lessons 

3 

3 

2 

7 

0 

9 

The  information  I  learned  will  help  me  perform  my 
job  better 

■ 

0 

2 

1 

12 

0 

10 

I  found  the  material  in  this  course  very  useful 

1 

0 

1 

5 

8 

1 

11 

Test  questions  were  based  upon  the  learning 
objectives 

5 

1 

0 

3 

6 

1 

12 

Test  questions  only  covered  items  I  had  been  taught 

5 

1 

2 

2 

4 

2 

13 

Directions  for  each  test  were  clear 

5 

0 

0 

3 

6 

2 

14 

I  understood  the  performance  standard  for  each  test 
or  quiz 

4 

0 

1 

2 

6 

3 

15 

I  understood  the  performance  standard  for  each 
graded  activity 

4 

0 

1 

2 

7 

2 

16 

Tests  were  fairly  and  properly  proctored  / 
administered 

10 

0 

0 

■ 

H 

■ 

17 

I  received  timely  feedback  about  my  test  scores 

E9 

0 

1 

2 

2 

0 

18 

The  feedback  I  received  reqardinq  tests  was  valuable 

H 

0 

0 

2 

3 

0 

19 

The  feedback  I  received  regarding  my  test  scores 
will  help  me  improve  my  performance 

H 

0 

0 

■ 

H 

0 

20 

Graded  activities  were  fairly  and  properly 
administered 

10 

0 

0 

3 

3 

0 

21 

I  received  timely  feedback  regarding  my  graded 
activities 

0 

0 

2 

3 

0 

22 

The  feedback  I  received  regarding  my  graded 
activities  was  valuable 

H 

0 

0 

2 

3 

0 

23 

The  feedback  I  received  regarding  my  graded 
activities  will  help  me  improved  mv  performance 

H 

0 

0 

3 

2 

0 

24 

The  software  was  easy  to  install 

25 

The  software  was  easy  to  use 

26 

The  software  menu  was  easy  to  find 

27 

The  software  was  easy  to  navigate 

28 

Directions  were  easy  to  find 

29 

Directions  were  easy  to  follow 

30 

On-screen  options  were  easily  understood 

31 

The  software  had  an  integrated  search  function 

32 

I  always  knew  where  to  look  for  on-screen  help 

33 

Human  help  was  always  available 

34 

I  was  always  able  to  page  forward 

35 

I  was  always  able  to  page  back 

36 

I  was  able  to  bookmark  my  location  and  come  back 
later 

37 

I  was  able  to  track  my  progress  through  each  lesson 

38 

I  was  able  to  track  my  progress  through  the  training 
program 

39 

I  was  able  to  easily  generate  practice  tests 

40 

Each  lesson  kept  my  attention 

41 

I  thought  the  lessons  were  entertaining 

42 

The  multimedia  used  in  each  lesson  was  relevant 

43 

Lessons  incorporated  audio  narration 

44 

High-resolution  graphics  /  pictures  were 

incorporated  into  each  lesson 

45 

The  use  of  relevant  video,  animation,  and  pictures 
were  incorporated  into  each  lesson 

46 

The  use  of  video,  animation,  and  pictures  in  each 
lesson  kept  my  interest 

47 

There  were  too  many  videos,  animations,  or  pictures 
in  each  lesson 

48 

The  information  in  each  lesson  was  worthwhile 

49 

The  courseware  taught  me  useful  information 

50 

After  using  the  courseware,  I  understand  the 
material  better 

51 

I  was  frequently  bored  while  using  the  instructional 
software 

52 

The  use  of  color  enhance  the  instructional 
presentations 

53 

Contrasts  in  the  use  of  color  made  text  easier  to 
read 

54 

I  always  had  access  to  a  quiet,  comfortable, 
distraction  free  learning  environment 

55 

I  was  frequently  interrupted  or  called  away  while 
using  the  courseware 

56 

I  was  able  to  start  and  complete  a  lesson  without 
interruption 

8  0 


0 


2  10 


7 

3 

0 

2 

0 

2 

0 

2 

0  0 
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57 

I  always  had  immediate  access  to  instructional 
resources  such  as  computers,  software,  books, 
instructions  or  manuals 

0 

8 

■ 

2 

2 

■ 

58 

A  different  location  can  or  should  be  used  for 
learninq  via  this  CBT  software 

■ 

■ 

2 

2 

6 

5 

59 

I  was  able  to  easily  read  textual  information 

2 

1 

0 

4 

9 

1 

60 

I  had  to  scroll  down  through  the  text  that  was  on  the 
screen 

4 

3 

5 

3 

2 

0 

61 

The  questions  I  was  asked  focused  on  information  I 
had  already  been  tauqht 

■ 

■ 

7 

H 

H 

0 

62 

The  questions  I  was  asked  focused  on  information 
that  I  was  about  to  learn 

■ 

0 

6 

3 

7 

0 

63 

The  questions  I  was  asked  were  related  to  the 
topic's  learning  objectives 

■ 

■ 

0 

1 

3 

1 

Table  2.  Responses  to  Student  Survey,  Part  II 


B.  RESPONSES  TO  SUPERVISOR  SURVEY,  PART  II 


NUMBER 

QUESTION 

N/A 

1 

2 

3 

4 

5 

1 

The  SWOSCOLCOM  Division  Officer  At  Sea  Training 
Initiative's  qoals  are  well  defined 

0 

0 

0 

1 

3 

0 

2 

I  know  where  the  SWOSCOLCOM  Division  Officer  At 
Sea  Training  Initiative's  goals  are  defined 

0 

0 

0 

3 

0 

3 

The  Training  Initiative's  goals  do  not  conflict  with 
other  organizational  goals 

0 

0 

■ 

■ 

2 

0 

4 

This  course  reduces  the  student's  OJT  requirements 

0 

3 

1 

0 

0 

0 

5 

This  course  reduces  the  time  it  takes  students  to 
learn  Surface  Warfare  Officer  job  skills 

0 

2 

2 

0 

0 

0 

6 

Each  topic  has  associated  learning  objectives 

0 

1 

2 

0 

1 

0 

7 

Learning  objectives  are  presented  at  the  beginning 
of  every  lesson 

0 

0 

0 

0 

4 

0 

8 

Learning  objectives  include  key  words  or  phrases 
such  as  create,  install,  set-up,  start,  repair, 
diagnose,  troubleshoot,  organize  or  write 

0 

0 

0 

0 

4 

0 

9 

Learning  objectives  include  key  words  or  phrases 
such  as  understand,  list,  explain,  give  an  example 
of,  describe,  identify,  recall,  master,  or  differentiate 

0 

0 

0 

1 

3 

0 

10 

The  learning  objectives  support  the  organizational 
training  initiative's  goals 

0 

0 

1 

2 

1 

0 

11 

Test  questions  are  derived  from  the  learning 
objectives 

0 

0 

0 

0 

3 

1 

12 

Students  performance  standards  are  derived  from 
the  learninq  objectives 

0 

0 

2 

■ 

■ 

0 

13 

All  student  performance  standards  are  published 

0 

0 

3 

1 

0 

0 

14 

Students  understand  the  grading  criteria  for  each 
and  every  test  and  graded  activity 

1 

0 

2 

0 

1 

0 

15 

Information  taught  to  the  students  is  relevant  to  the 
topic 

0 

0 

1 

0 

2 

1 
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16 

The  instructional  presentations  cover  all  the  learning 
obiectives 

0 

0 

17 

Instructional  material  is  current  and  up  to  date 

0 

0 

18 

Instructional  material  will  help  the  students  perform 
their  jobs  better 

0 

0 

19 

Directions  for  each  test  and  qraded  activity  are  clear 

1 

0 

20 

Tests  and  graded  activities  are  fair 

1 

0 

21 

Remediation  is  available  for  students 

0 

0 

22 

Remediation  for  failing  to  meet  a  standard  is 
effective 

0 

0 

23 

Each  test  proctor  or  activity  participant  other  than 
the  student  is  identified  in  advance 

0 

0 

24 

Each  test  proctor  or  activity  participant  other  than 
the  student  is  trained  and  qualified  for  the  part  of 
the  test  or  activity  they  are  participating  in 

0 

0 

25 

All  tests  are  given  and  graded  by  the  computer 

2 

0 

26 

For  all  test  items  not  graded  by  a  computer,  the 
criteria  for  a  fully  correct  answer  is  known  by  the 
grader 

2 

0 

27 

For  all  test  items  not  graded  by  a  computer,  the 
performance  criteria  for  fully  correct  answers  are 
established 

2 

0 

28 

For  all  test  items  not  graded  by  a  computer,  the 
criteria  for  a  fully  correct  answer  is  used  to  grade  a 
particular  test  item 

2 

0 

29 

For  all  test  items  not  graded  by  a  computer,  the 
same  person  that  writes  the  test  also  grades  it 

2 

0 

30 

For  all  performance  based  activities,  the  criteria  for 
correctly  performed  actions  is  known  by  the  grader 

3 

0 

31 

For  all  performance  based  activities,  the  criteria  for 
correctly  performed  actions  is  always  strictly  used 
by  the  grader 

2 

0 

32 

The  collection  of  student  test  and  activity 
performance  data  enhances  organizational 

performance 

2 

0 

33 

Students  receive  timely  feedback  regarding  their 
performance 

2 

0 

34 

Feedback  students  receive  is  relevant  and  useful  to 
them 

2 

0 

35 

Feedback  to  students  either  praises  them  or 
identifies  a  weakness  and  reveals  how  to  correct  the 
deficiency 

2 

0 

36 

The  software  /  courseware  is  easy  to  install 

0 

3 

37 

The  software  /  courseware  is  easy  to  use 

0 

2 

38 

Student  progress  is  easily  tracked  via  the 
courseware 

0 

3 

39 

The  courseware  adequately  protects  personal 
information 

0 

0 

40 

Files  where  personal  or  performance  information  is 
stored  are  password  protected 

0 

0 

3 

0 

3 

0 

0  0  3 


0  0  2  0  0 


0  0  3 


0  0 


0  0  2  0  0 


0 


2  0 


0  2 


0  0 


0  0 


0  2  0  0  0 


0  0  0  2  0 


0  0 


0  0 


0  0 


0  0 
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41 

Personal  and  performance  information  is  encrypted 
durinq  transfers  between  locations 

0 

0 

0 

4 

0 

0 

42 

Personal  and  performance  data  is  stored,  updated, 
and  maintained  on  a  computer  that  has  direct 
access  to  the  Internet 

0 

0 

0 

■ 

3 

0 

43 

Personal  and  performance  data  is  only  downloaded 
on  a  computer  that  has  direct  access  to  the 
Internet,  but  is  stored  on  a  computer  that  has  zero 
Internet  access (air  gapped) 

0 

0 

1 

0 

0 

0 

44 

Students  are  provided  a  quiet,  distraction  fee 
environment 

0 

0 

2 

0 

■ 

■ 

45 

Students  are  not  interrupted  or  called  away  while 
using  the  courseware  to  learn 

0 

1 

2 

0 

1 

0 

46 

A  different  location  can  or  should  be  used  for 
learninq  via  this  computer-based  traininq  software 

0 

0 

0 

3 

0 

47 

This  computer-based  training  program's 

administrators  and  supervisors  are  all  adequately 
trained 

0 

3 

■ 

0 

0 

0 

48 

Evaluation  standards  for  assessing  this  computer- 
based  training  initiative  have  been  established 

0 

0 

3 

0 

1 

0 

49 

Evaluation  standards  for  assessing  this  computer- 
based  training  initiative  have  been  promulgated 

0 

0 

3 

1 

0 

0 

50 

The  standards  for  this  computer-based  training 
initiative  are  being  met  or  exceeded 

■ 

■ 

2 

0 

0 

0 

51 

Formal  standards  are  established  for  the 
presentation  of  this  training  initiative's  instructional 
material  by  means  other  than  the  computer  (watch 
standers) 

0 

0 

2 

0 

2 

0 

52 

Instructional  presentations  of  this  training 
initiative's  material  by  means  other  than  a  computer 
is  closely  monitored 

0 

0 

2 

0 

2 

0 

Table  3.  Responses  to  Supervisor  Survey,  Part  II 

C.  RESPONSES  TO  STUDENT  SURVEYS,  PART  III 

Question  1.  Have  you  ever  failed  to  meet  the  required  standards  for  an 
activity  or  test?  If  yes,  why  do  you  believe  that  you  failed  to  meet  the 
required  standards? 

(1)  Yes,  sometimes  test  /  quiz  questions  asked  for  material  that  was  not 
sufficiently  covered  during  the  lesson. 

(2)  Yes,  DC  was  especially  difficult.  The  test  (practical  test)  asked  many 
questions  that  weren’t  in  the  training. 
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(3)  Only  on  the  practical  assessments.  I  believe  the  cause  was  too  much 
information  to  go  through  before  being  tested.  It  was  hard  to  retain  the  info(4) 
Yes,  the  required  standards  were  never  stated. 

(5)  Yes  for  an  exercise,  no  for  test. 

(6)  Yes,  the  exams  are  too  detailed  with  useless  information. 

(7)  I  have  not  attempted  a  test  yet. 

(8)  I  have  passed  all  tests  taken. 

Question  2.  If  you  answered  yes  to  the  above  question,  did  you  receive 
remediation?  If  yes,  why  was  the  remediation  you  received  either  effective 
or  ineffective? 

(1)  No,  there  is  no  sort  of  remediation  system  on  this  ship. 

(2)  No... no  support  resources  were  available.  Our  DC  workbooks,  for  example, 
leaves  a  lot  to  be  desired. 

(3)  Never  received  remediation  because  the  administration  did  not  know  the 
standards,  therefore  it  was  ineffective. 

(4)  Received  immediate  remediation  for  the  exercise  I  failed  to  meet 
requirements  for.  Very  effective. 

Question  3.  What  should  be  done  to  improve  this  course  of  instruction? 

(1)  Make  a  conscious  consistent  effort  to  upgrade  the  software. 

(2)  More  computers 

(3)  Give  us  a  training  manual.  Give  us  our  own  copy  of  the  software  if  you  don’t 
give  us  a  training  manual  or  give  us  a  computer  to  install  it  on. 

(4)  There  should  be  a  room  or  place  where  JO’s  can  work  on  their  training.  I 
have  no  place  to  do  my  computer  training. 

(5)  Reduce  the  number  of  practicums  and  possibly  make  them  more 
consolidated.  There  is  plenty  of  ways  to  get  the  experience  and  knowledge 
without  having  to  jump  through  hoops  in  order  to  get  stuff  done.  Allowing  more 
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time  to  work  on  the  training  would  be  a  good  help.  I  don’t  think  a  division  should 
be  given  a  new  JO  right  away.  Give  him/her  time  to  establish  a  good  routine  and 
to  get  accustomed  to  the  program  before  having  to  run  a  division  as  well. 

(6)  Create  an  environment  where  there  are  computers  and  time  to  do  it.  Make 
installation  easier.  Create  reasonable  tests  that  test  common  knowledge  that  we 
will  remember. 

(7)  Handbooks  and  slightly  clearer  instructions  on  how  to  maneuver  the  modules 
and  course  information. 

(8)  Issue  every  Ensign  a  computer  and  a  set  of  discs.  Allow  more  OJT  in 
conjunction  with  the  modules.  Shorten  the  modules  or  increase  the  amount  of 
lessons  to  decrease  the  amount  of  time  spent  learning.  Train  the  trainers.  Give 
them  a  course  explaining  the  program  and  how  to  best  implement  it. 

(9)  Utilize  decommissioning  FFG’s  and  DD’s  as  “school  ships”  to  maximize  the 
hands-on  training  and  dedicated  training  personnel. 

(10)  There  is  no  doubt  that  much  thought  was  put  into  this  program... the  only 
criticism  I  have  lies  in  the  program’s  organization  and  execution.  There  is  no 
doubt  that  ships  are  not  being  given  the  technology  equipment  and  guidance  on 
how  to  carry  out  the  programs  intent.  Training  Officers  are  not  given  training  on 
how  to  carry  out  the  program.  With  time,  it  will  surely  improve. 

(11)  Send  us  back  to  SWOS.  I  believe  it  is  a  needed  part  of  officer  training-just 
shorten  it  some.  Until  the  Navy  can  dedicate  a  person  to  teaching  SWOS-at-Sea 
on  each  ship  and  train  that  person  first  on  how  to  run  the  program,  the  program 
will  not  work.  Then  ships  need  the  funding  to  run  this;  i.e.  computers  for  the 
program  and  a  space  to  work  in.  To  make  this  program  really  effective,  the  Navy 
needs  to  put  this  program  at  the  top  of  the  list  for  all  new  Ensigns  and  their  CO’s. 
As  long  as  the  Ensigns  have  command  duties,  the  program  will  never  be 
effective.  My  personal  feeling  is  that  new  officers  with  no  prior  military 
experience  need  something  prior  to  the  ship  that  is  military  in  manner. 
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(12)  We  need  more  structured  instruction  and  dedicated  personnel.  Ship’s 
officers  are  too  busy  with  other  primary  duties. 

(13)  Ensure  proper  training  of  system  administrators/trainers.  Provide  computers 
devoted  entirely  to  the  SWOS-at-Sea  program.  Overhaul  some  of  the  lessons  so 
they  provide  more  relevant  info  in  a  more  engaging  format.  Overhaul  the  tests  so 
that  they  evaluate  the  material  that  was  presented  in  the  lessons.  Solve 
technical  problems  with  the  program. 

(14)  The  program  should  be  reconstructed  to  be  more  user  friendly.  Without  the 
help  of  someone  helping  me  with  the  initial  startup  process  I  am  unable  to  start 
the  program  at  all. 

Question  4.  What  are  the  best  parts  of  this  course  and  why? 

(1)  I  have  little  experience  with  the  course,  but  I  like  the  simplicity. 

(2)  A  lot  of  the  material  is  pertinent,  and  beneficial.  It  is  good  to  be  able  to  study 
this  material  on  your  own.  Some  to  the  multimedia  is  interesting  and  helpful. 
Combining  formalized  instruction  via  the  CD  and  the  OJT  is  probably  more 
effective  than  6  months  in  a  classroom. 

(3)  The  information  is  very  good,  and  it  does  reinforce  our  practical 
understanding.  However,  standing  5  and  dimes  (A  type  of  watch  section  rotation 
where  a  watch  stander  typically  stands  five  hours  of  watch  and  then  has  ten 
hours  off  before  coming  back  on  watch  again)  underway,  it  is  difficult  to  summon 
one’s  energy  to  find  computer  time  and  complete  lessons.  We  are  not  supposed 
to  be  division  officers,  and  I  am  taking  the  responsibility  for  a  lot  of  divisional 
duties  because  the  opportunity  is  there. 

(4)  Some  of  the  lessons  have  good  visuals  that  help  the  learning  process  and 
could  not  be  accomplished  with  a  book  or  chalkboard.  Could  be  a  good  way  to 
learn  if  nothing  else  was  going  on. 

(5)  Interactive  videos,  sounds,  questions.  They  keep  my  attention  better  than 
just  textual  slides. 

(6)  Saves  money.  Immediate  utilization  of  bodies. 
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(7)  The  slides  are  very  informative  and  contain  useful  information.  The 
knowledge  is  presented  in  a  plain  and  easily  understood  manner.  The  animation 
in  the  later  slides  add  an  element  of  excitement  to  the  lessons  as  well  as  make  it 
easier  to  pay  attention  and  envision  real  life  application  of  knowledge. 

(8)  You  don’t  have  to  go  to  SWOS.  Good  way  of  getting  information  without 
asking  people. 

(9)  Learn  at  own  pace  and  ability  to  work  when  I  have  time.  It  allows  flexibility  in 
the  schedule  while  still  teaching  me  important  information.  The  info  did  improve 
my  knowledge  of  the  ship  and  its  abilities  while  seeing  them  first  hand.  I  know  it 
makes  a  good  learning  experience. 

(10)  The  information  is  helpful. 

Question  5.  What  are  the  worst  parts  of  this  course  and  why? 

(1)  I  have  difficulty  saving  and  progress.  I  don’t  know  if  I  am  doing  it  wrong  or 
there  is  a  real  problem. 

(2)  Technical  problems  with  the  program.  A  lack  of  guidance  /  instruction.  NO 
COMPUTER  AVAILABILITY;  most  of  ship’s  computers  being  used  for  ships 
work.  Some  of  the  slides  are  too  long  and  boring... not  presented  in  an  engaging 
format.  Some  of  questions  on  the  quizzes  /  tests  asked  for  information  that  was 
not  adequately  presented  in  the  lessons.  The  SWOS-at-sea  program  is  not  a 
priority  for  most  ships... it  is  difficult  to  find  officers  who  have  enough  time  to 
review  the  material  thoroughly. 

(3)  We  need  dedicated  computer  time  and  space.  Dedicated  instructors  are 
needed... i.e.  shore  based  learning  center  (ensuring  fairness).  Too  many  officers 
so  we  don’t  actually  get  enough  driving  /  learning  time. 

(4)  No  human  interaction  for  ready  reference  questions.  The  programs  often 
don’t  work.  Ships  cannot  dedicate  time  to  working  on  this... there  are  always 
other  things  going  on  that  are  mission  critical  for  the  ship  and  they  take  priority.  It 
is  very  difficult  to  dedicate  time  to  learn  when  your  have  10-12  hours  of  watch,  4 
hours  of  meetings,  and  various  other  things  going  on  with  little  or  no  sleep  (A 
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normal  underway).  Underway  is  a  bad  environment  for  sit  down  learning.  Some 
functions  are  hard  to  operate  or  don’t  work  at  all.  To  date  (over  3  months),  our 
testing  CD’s  are  not  working.  Other  ships  have  reported  that  the  tests  are  not 
saving  upon  completion. 

(5)  Length  of  lessons.  Mundane  textual  slides.  Difficult  to  maintain  attention 
span,  especially  while  underway  and  standing  watches  with  little  sleep. 

(6)  IT  problems  are  rampant.  The  modules  will  not  open  or  information  will  not 
save.  Standards  were  never  explained  to  the  trainees,  therefore  performance  is 
low.  The  ship  was  never  trained  in  how  the  program  should  run.  There  are  no 
dedicated  trainers;  therefore  students  are  not  a  priority.  The  lack  of  training  is 
not  complimentary  to  the  profession.  Over-manning  decreases  hands-on  training 
at  tasks  such  as  ship  driving. 

(7)  Sitting  at  a  computer  reading  for  hours  is  very  boring.  The  modules  are  very 
long  and  require  a  lot  of  time  to  complete.  Procuring  a  computer  for  a  sufficient 
amount  of  time  to  complete  a  module  and  learn  the  material  was  very  difficult. 
Learning  is  not  a  priority  onboard  and  outside  resources  were  not  readily 
available.  OJT  is  very  limited. 

(8)  Loss  of  human  interaction  leads  to  a  loss  of  initiative.  Hard  to  stay  focused 
and  can’t  ask  questions  on  specific  problems.  It  is  very  hard  to  balance  the  Divo 
at  sea  courseware  requirements  and  qualifications.  Qualifications  tend  to  take  a 
majority  of  the  time  which  leaves  less  time  with  the  courseware. 

(9)  There  are  too  many  slides  and  too  much  information  to  do  while  being  a  Divo 
and  also  learning  the  same  stuff  in  your  PQS.  If  you  don’t  have  a  laptop  you 
can’t  complete  the  computer  training.  Installation  needs  to  be  put  into  a  single 
.exe  file  which  is  extractable  with  one  click. 

(10)  There  are  too  many  activities  (practicums).  While  I  understand  the 
relevance  and  importance,  I  frequently  find  myself  lacking  enough  time  to  work 
on  them.  Balancing  a  division,  watches,  other  PQS,  and  other  aspects  of  ship 
life  doesn’t  leave  much  time  to  run  around  looking  up  references  and  observing 
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evolutions,  as  well  as  cross  decking  to  other  ships  to  see  how  they  do  it.  Just  too 
much  to  do  in  too  little  time. 

(11)  The  length  of  each  module  and  how  interesting  the  lessons  are. 

(12)  I  would  like  hard  copy,  like  a  textbook  or  (reference)  TRAMAN  for  learning 
the  material 

(1 3)  Give  each  person  a  set  of  discs. 

D.  RESPONSES  TO  SUPERVISOR  SURVEY,  PART  III 

Question  1.  If  the  SWOSCOLCOM  Division  Officer  at  Sea  Training 
Initiative’s  goals  can  be  better  defined,  how  would  you  change  them  to 
make  better? 

(1)  When  the  system  actually  comes  up  on  the  computer,  the  goals  are  easy  to 
identify. 

(2)  Better  printed  reference  material  should  be  provided. 

Question  2.  How  are  the  students’  performance  standards  for  graded 
activities  established?  Are  these  standards  uniform  throughout  the  fleet? 

(1 )  The  results  /  answers  are  given  through  the  program. 

Question  3.  What  is  done  with  the  collected  student  test  and  activity 
performance  data? 

(1 )  Sent  off  to  SWOS.  Nobody  has  taken  a  test  yet. 

Question  4.  Does  the  collection  of  student  test  and  activity  performance 
data  enhance  or  benefit  organizational  performance?  If  so,  how? 

(1)  Can’t  tell  yet  because  nobody  has  taken  a  test.  The  Ensigns  are  excelling  in 
SWO  knowledge  by  OJT. 

Question  5.  Does  the  organization  require  periodic  evaluations  of  this 
training  initiative?  If  so,  what  aspects  of  this  Computer-based  training 
initiative  are  to  be  evaluated? 

(1)  Everything  needs  to  be  evaluated. 
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Question  6.  What  is  the  procedure  for  reporting  weaknesses  or 
deficiencies  that  are  discovered  in  the  SWOSCOLCOM  Division  Officer  at 
Sea  Training  Initiative? 

(1)  Via  Email. 

(2)  Email  SWOS  POC. 

Question  7.  If  a  weakness  or  deficiency  is  discovered  in  eh  SWOSCOLCOM 
Division  Officer  at  Sea  Training  Initiative,  what  is  the  maximum  time  that  is 
allowed  to  correct  it? 

(1)  Until  it  gets  resolved. 

(2)  The  problem  is  identified.  If  computer  disc/software  discrepancies  or  missing 
data,  and  turn  around  time  until  receipt  of  correction  CD  was  around  2  to  3 
months. 

Question  8.  Are  standards  established  for  the  presentation  of  instructional 
material  via  CBT?  If  yes,  please  explain  why  or  how  those  standards  have 
or  have  not  been  met. 

(1)  No,  zero  guidance  was  given  about  the  program  and  since  it  is  difficult  to  set 
up  on  the  LAN,  we  don’t  have  a  presentation  of  instructional  material.  It’s  all  over 
the  LAN  and  difficult  to  navigate  through. 

(2)  No  specific  guidelines  have  been  established  as  of  yet.  Students  are 
expected  to  review  modules  on  their  own  time  with  the  primary  responsibilities 
still  being  watch  stander  qualification  and  divisional  obligations. 

Question  9.  What  is  /  are  the  major  strengths  of  this  CBT  initiative? 

(1)  Appears  as  though  decent  information  is  on  the  discs  once  we  can  pull  them 
up. 

(2)  The  concept  of  qualifying  SWOS  more  quickly. 

(3)  Information/diagrams  are  very  useful  in  illustrating  certain  theories  or  systems 
(e.g.  damage  control,  engineering,  combat  systems)  and  have  been  used  to 

garner  info  for  larger  command  wide  officer  training. 
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Question  10.  What  is  /  are  the  major  weaknesses  of  this  CBT  initiative? 

(1)  Difficult  to  navigate  through.  No  instructions  were  given  to  the  training 
officers.  It  was  just  sent  out  to  the  fleets.  Looks  like  the  product  had  not  been 
QA’d  because  many  modules  were  missing  or  incomplete.  No  real 
troubleshooting  help  is  available.  Not  much  guidance. 

(2)  The  printed  reference  material  is  poor.  The  installation  is  poorly  explained. 
The  support  to  the  ships  is  weak. 

(3)  Command  shortage  of  available  computer  resources  to  dedicate  to  solely 
CBT  program.  Personal  aspect  eliminated-most  instruction  comes  from  CD’s 
and  questions  or  OJT  issues  cleared  up  while  on  watch.  Loading  CD’s  and 
tracking  current  versions  while  deployed  was  difficult.  Command  emphasis  more 
towards  PQS  program  rather  than  CBT  now  that  SWOS  has  been  essentially 
taken  out  of  the  picture. 

Question  1 1 .  What  should  be  done  to  improve  this  course  of  instruction? 

(1)  Instruct  the  supervisors  before  sending  out  the  material.  Have  teams  come  to 
the  ship  and  implement  it  on  the  LAN  and  then  give  lectures  on  how  to  navigate 
through.  Explain  saving  and  exporting  data  procedures. 

(2)  The  CBT  lacks  Polish.  Installation  is  a  pain  and  there  is  not  any  real  tech 
support.  The  CBT  is  not  geared  to  the  ship’s  LANs.  Have  had  a  hard  time 
finding  computers  that  can  run  the  program.  We  have  had  a  hard  time  finding 
computers  with  the  appropriate  software  and  hardware. 

(3)  Training  seminar  on  the  waterfront  for  implementation  and  expectations  of 
program.  With  this  program  coming  into  the  fleet  during  the  beginning  of  the  Gulf 
War,  information  and  program  instructions  were  difficult  to  ascertain.  Hopefully, 
a  waterfront  training  initiative  would  get  everyone  on  the  same  page. 
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APPENDIX  C.  HISTORICAL  SWOS  DATA 


A.  SWOS  DOC  TOPICS  AND  THEIR  RELATIVE  IMPORTANCE 


Topic 

Number 

Topic  Title 

Survey 

Score 

Survey 

Ranking 

1.01 

Review  Preliminary  Rules  of  the  Road 

3.840 

1 

2.04 

Standard  Commands 

3.811 

2 

2.02 

R/T  Procedures 

3.809 

3 

5B.4 

Opening  CPA/  Avoiding  Course 

3.809 

4 

1.03 

Steering  and  Sailing  Rules 

3.784 

4 

5B.1 

Introduction  to  Maneuvering  Boards 

3.779 

5 

5A.5 

UNREP 

3.747 

6 

5B.2 

Tracking 

3.747 

7 

2.19 

Emergency  Actions 

3.730 

8 

1.02 

Lights  and  Dayshapes 

3.722 

9 

2.17 

Man  Overboard  Procedures 

3.716 

10 

5A.13 

Aids  to  Navigation 

3.716 

11 

1.04 

Restricted  Visibility  Sound  and  conduct 

3.711 

12 

11.18 

Firefighting  School 

3.710 

13 

5A.10 

Charts  and  Publications 

3.684 

14 

5A.20 

GPS 

3.684 

15 

2.11 

Shiphandling  Alongside  a  Pier 

3.670 

16 

2.12 

CONREP  Conning 

3.660 

17 

5A.19 

Navigation  Detail  and  Anchoring 

3.650 

18 

5A.2 

Mooring 

3.631 

19 

10.01 

3-M  and  the  Division  Officer 

3.628 

20 

11.17 

Buttercup  Get  Wet  Trainer 

3.622 

21 

5B.3 

Changing  of  Station 

3.621 

22 

5A.1 

Deck  Seamanship 

3.600 

23 

10.05 

Equipment  Tag-Out 

3.595 

24 

11.12 

Personnel  Protective  Equipment 

3.590 

25 

2.05 

Communications  and  the  Watchstander 

3.560 

26 

Portable  DC  Equipment 

3.550 

27 

11.07 

Fires  and  Extinguishing  Agents 

3.540 

28 

11.09 

Fixed  DC  Systems 

3.532 

29 

5B.8 

True/Desired  Wind 

3.526 

30 

6.13 

Electrical  Safety 

3.526 

31 

3.04 

Honors  and  Ceremonies 

3.490 

32 

4.12 

Navy  Safety  Programs 

3.474 

33 

10.07 

Quarterly  and  Weekly  Schedules 

3.473 

34 

5A.6 

Flight  Deck 

3.463 

35 

6.06 

PQS  System  With  Practical 

3.457 

36 

5B.5 

Tactics,  Columns,  and  Screen  Formations 

3.450 

37 

7.09 

Tactical  Data  Links 

3.426 

38 

11.02 

Compartmentalization  and  WTI 

3.426 

39 

3.03 

Ship's  Emergencies/Security  Threats 

3.420 

40 

6.02 

IDTC  and  Division  Officer  Inspections 

3.415 

41 

6.1 

Casualty  Reports 

3.409 

42 

11.16 

Practical  DC 

3.396 

43 

5A.12 

Tides  and  Currents 

3.390 

44 

10.06 

Cycle  Schedules 

3.383 

45 

11.15 

DC  Communications  and  Symbology 

3.380 

46 

6.08 

Message  Formatting  ADP 

3.376 

47 

11.19 

CBR-D:  Protective  Equipment 

3.374 

48 

11.2 

CBR-D:  Chemical,  Biological,  Radiological 

3.374 

49 

11.05 

Portable  Dewatering  Equipment 

3.372 

50 

3.05 

Small  Boats 

3.367 

51 

11.06 

Shoring,  Plugging,  and  Patching 

3.362 

52 

8.05 

U.S.  Aircraft  and  Weapons 

3.362 

53 

8.02 

U.S.  Combatants 

3.362 

54 

2.09 

Bridge  Equipment 

3.358 

55 

5A.3 

Anchoring  and  Mooring  to  a  Buoy 

3.358 

56 

2.01 

Surface  Combat  Information  Center 

3.351 

57 

4.05 

JO  Fitness  Reports  and  Enlisted  Evaluations 

3.347 

58 

5B.6 

Line  Formations 

3.337 

59 

2.07 

Underway  Bridge  Watch  Relief 

3.333 

60 

8.03 

Amphibious  Ships  and  Craft 

3.330 

61 

11.01 

Intro  and  Basic  First  Aid 

3.323 

62 

5A.11 

Basics  of  Time 

3.305 

63 

10.03 

MDS  and  the  Division  Officer 

3.298 

64 

5A.14 

Compasses 

3.295 

65 

4.02 

Enlisted  Rating  Structure  and  Advancement 

3.287 

66 

8.16 

Surface  Warfare 

3.287 

67 

2.08 

ATP-1  &  Publication  102 

3.284 

68 

2.03 

Underway  Watch  Organization 

3.284 

69 

11.14 

DC  Organization  and  Administration 

3.280 

70 

7.08 

NTDS/CDS/DTE 

3.280 

71 

11.1 

DCA  School  Lab  Tour 

3.272 

72 

6.04 

Shipboard  Training  Program 

3.269 

73 

8.06 

U.S.  Submarines 

3.269 

74 

4.01 

Division  Officer  Indoctrination 

3.256 

75 

7.02 

Shipboard  Communications  Systems 

3.255 

76 

8.19 

VBSS 

3.247 

77 

3.02 

Routine/special  Evolutions 

3.240 

78 

3.01 

Inport  Watch  Organization 

3.231 

79 

4.04 

Sexual  Harassment/Fraternization/Hazing 

3.221 

80 

8.04 

Auxiliaries 

3.215 

81 

5A.7 

Survival  at  Sea 

3.213 

82 
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8.17 

Air  Warfare 

3.213 

83 

6.12 

Operational  Messages 

3.210 

84 

7.03 

Shipboard  Satellite  Communication  Systems  & 
Equipment 

3.200 

85 

5A.21 

The  Future  of  Electronic  Navigation 

3.200 

86 

7.14 

Launching  Systems 

3.190 

87 

6.01 

Information  Security 

3.180 

88 

7.16 

Cruise  Missiles 

3.170 

89 

7.17 

Air  Warfare  (AW)  Missiles 

3.170 

90 

2.14 

ATP  &  Publication  102  Assignment 

3.160 

91 

8.18 

Undersea  Warfare 

3.160 

92 

8.08 

Rules  of  Engagement 

3.150 

93 

7.15 

Guided  Weapons 

3.130 

94 

4.03 

Enlisted/Officer  Service  Records 

3.090 

95 

6.03 

Schools  and  Exercises  With  Practical 

3.090 

96 

7.1 

MK  15  CIWS 

3.090 

97 

8.07 

Intro  to  Command  and  Control  Systems 

3.080 

98 

2.18 

Navy  Publications 

3.070 

99 

7.19 

Sonar  Fundamentals 

3.040 

100 

7.06 

IFF/TACAN 

3.020 

101 

7.18 

AEGIS  Weapon  System 

3.020 

102 

7.23 

USW  Weapons 

3.020 

103 

7.04 

Radar  Fundamentals  and  Systems 

3.010 

104 

11.13 

Basic  Gas  Free  Engineering 

3.010 

105 

5A.27 

Electronic  Navigation  Practical 

3.010 

106 

2.06 

R/T  Crypto  System 

3.000 

107 

2.16 

Special  CIC  Evolutions 

2.990 

108 

7.05 

Introductions  to  Electronic  Warfare 

2.990 

109 

7.13 

Naval  Gun  Weapon  Systems 

2.990 

110 

8.12 

Intro  to  Amphibious  Operations 

2.990 

111 

5A.18 

Sunrise  and  Sunset 

2.990 

112 

4.06 

Drug  and  alcohol  Abuse 

2.980 

113 

11.04 

Stability 

2.970 

114 

4.07 

Good  Order  and  Discipline/Shore  Patrol 

2.950 

115 

4.08 

Manpower  Documents 

2.950 

116 

5B.7 

Search  Turns 

2.950 

117 

6.05 

C/S  Inport  Training  Program 

2.930 

118 

5A.4 

Towing 

2.930 

119 

8.14 

Mine  Warfare 

2.880 

120 

8.15 

Amphibious  Assault 

2.870 

121 

10.04 

Alterations  and  Availabilities 

2.870 

122 

6.15 

CSOSS 

2.860 

123 

7.07 

The  Gunfire  Control  Problem 

2.860 

124 

7.2 

Acoustic  Propagation 

2.850 

125 

4.13 

Non-Judicial  Punishment 

2.820 

126 
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7.12 

Naval  Ammunition 

2.800 

127 

7.21 

AN/SQQ-89(V)  Part  I 

2.800 

128 

2.15 

Visual  Communications 

2.770 

129 

7.22 

AN/SQQ-89(V)  Part  II 

2.760 

130 

8.01 

Naval  Doctrine 

2.750 

131 

2.13 

Subsurface  Combat  information  Center 

2.730 

132 

2.1 

Air  Combat  Information  Center 

2.710 

133 

4.09 

Transfers  and  Separations 

2.690 

134 

8.09 

Command  and  Control  Warfare 

2.680 

135 

7.01 

Radio  Wave  Propagation 

2.650 

136 

6.14 

Search  and  Seizure 

2.590 

137 

7.24 

Ship's  Silencing  Program 

2.570 

138 

8.11 

Marine  Corps 

2.450 

139 

4.1 

Absentees  and  Deserters 

2.390 

140 

1.98 

Unit  1  Exam 

No  Useful 
Data 

141 

1.99 

Unit  1  Exam  Review 

No  Useful 
Data 

142 

2.2 

Communications  Practical 

No  Useful 
Data 

143 

2.98 

Module  2/3  Exam 

No  Useful 
Data 

144 

3.98 

Unit  3  Exam 

No  Useful 
Data 

145 

4.11 

Family  Care  Policy  and  Pregnancy 

No  Useful 
Data 

146 

4.98 

Unit  4  Exam  (Take  Home) 

No  Useful 
Data 

147 

4.99 

Unit  4  Exam  Review 

No  Useful 
Data 

148 

6.07 

GENADMIN  Messages 

No  Useful 
Data 

149 

6.09 

Messaqe  Writinq  Practical 

No  Useful 
Data 

150 

6.11 

CASREP  Practical 

No  Useful 
Data 

151 

6.98 

Unit  6  Exam  (Take  Home) 

No  Useful 
Data 

152 

6.99 

Unit  6  Exam  Review 

No  Useful 
Data 

153 

7.97 

Unit  7  Exam  Preview 

No  Useful 
Data 

154 

7.98 

Unit  7  Exam 

No  Useful 
Data 

155 

7.99 

Unit  7  Exam  Review 

No  Useful 
Data 

156 

8.1 

Electronic  Warfare 

No  Useful 
Data 

157 

8.13 

Naval  Surface  Fire  Support 

No  Useful 
Data 

158 
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8.2 

Intelligence  Brief 

No  Useful 
Data 

159 

8.98 

Unit  8  Exam 

No  Useful 
Data 

160 

8.99 

Unit  8  Exam  Review 

No  Useful 
Data 

161 

10.02 

Supply  and  the  Division  Officer 

No  Useful 
Data 

162 

10.08 

PMS  Chanqe  Manual  and  Spot  Check 

No  Useful 
Data 

163 

10.12 

Automated  MDS  I 

No  Useful 
Data 

164 

10.16 

MDS/PMS  Practical  Review 

No  Useful 
Data 

165 

10.98 

Unit  10  Exam 

No  Useful 
Data 

166 

10.99 

Unit  10  Practical  Review 

No  Useful 
Data 

167 

11.03 

Firemain  and  Drainaqe  Systems 

No  Useful 
Data 

168 

11.08 

The  AFFF  System  and  Maqazine  Sprinklers 

No  Useful 
Data 

169 

11.97 

Unit  11  Review  for  Test 

No  Useful 
Data 

170 

11.98 

Unit  11  Exam 

No  Useful 
Data 

171 

11.99 

Unit  11  Exam  Review 

No  Useful 
Data 

172 

12.06 

Simulator  Indoctrination 

No  Useful 
Data 

173 

12.07 

DRT  and  Scopehead  Plottinq 

No  Useful 
Data 

174 

5A.15 

Piloting  I 

No  Useful 
Data 

175 

5A.16 

Pilotinq  II 

No  Useful 
Data 

176 

5A.17 

Piloting  III 

No  Useful 
Data 

177 

5A.25 

Navigation  Practical  I 

No  Useful 
Data 

178 

5A.26 

Navigation  Practical  II 

No  Useful 
Data 

179 

5A.98 

Unit  5A  Exam 

No  Useful 
Data 

180 

5A.99 

Unit  5A  Exam  Review 

No  Useful 
Data 

181 

5B.10 

Maneuvering  Board  Practical  Review 

No  Useful 
Data 

182 

5B.11 

Maneuvering  Board  Pre-test 

No  Useful 
Data 

183 

5B.12 

Unit  5B  Exam 

No  Useful 
Data 

184 
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5B.13 

Unit  5B  Exam  Review 

No  Useful 
Data 

185 

5B.9 

Maneuvering  Board  Practical 

No  Useful 
Data 

186 

Table  4.  Importance  Ratings  of  Topics  from  IDSI  Final  Report  (pp.  HI  -H24) 

B.  AVERAGE  NUMBER  OF  FAILURES  PER  CLASS  (NON  ENGINEERING) 

The  information  below  was  taken  directly  from  the  Microsoft  Excel  files 
maintained  at  SWOS.  This  information  does  not  include  data  from  the 
engineering  curriculums  (Core  Phase  II). 


Sample  Number 

CLASS 

NUMBER 

Unit  1 

Unit  2/3 

Unit  4 

Unit  5 

Unit  6 

1 

109 

7 

12 

0 

25 

0 

2 

110 

17 

4 

0 

34 

5 

3 

111 

24 

33 

0 

41 

7 

4 

112 

0 

2 

0 

9 

0 

5 

113 

0 

29 

2 

18 

9 

6 

114 

18 

29 

1 

32 

7 

7 

115 

17 

28 

2 

31 

34 

8 

116 

28 

10 

1 

45 

5 

9 

117 

5 

3 

2 

27 

5 

10 

118 

9 

14 

3 

34 

3 

11 

119 

24 

2 

3 

26 

13 

12 

120 

15 

13 

5 

72 

4 

13 

121 

9 

20 

2 

77 

10 

14 

122 

Unavailable 

Unavailable 

Unavailable 

Unavailable 

Unavailable 

15 

123 

12 

31 

12 

59 

6 

16 

124 

22 

11 

21 

74 

5 

17 

125 

32 

9 

13 

47 

1 

18 

126 

11 

7 

8 

38 

2 

19 

127 

22 

10 

9 

33 

1 

20 

128 

14 

15 

9 

18 

2 

21 

129 

24 

7 

2 

32 

13 

22 

130 

16 

4 

2 

21 

4 

23 

131 

63 

6 

9 

24 

4 

24 

132 

19 

6 

1 

24 

2 

120 


121 


Totals: 


Ave  Number  of 
Failures  per  class: 


Ave  Class  Failures 
as  a  Percentage: 


Sample  Number 


1 


2 


3 


Ave  Number  of 
Failures  per  class: 


Ave  Class  Failures 
as  a  Percentage: 


CLASS 

NUMBER 


109 


110 


111 


112 


113 


114 


115 


116 


117 


118 


119 


120 


121 


122 


123 


124 


125 


126 


127 


128 


129 


130 


131 


132 


133 


134 


135 


136 


137 


138 


Totals: 


642 

628 

650 

261 

319 

22.138 

21.655 

22.414 

9.000 

11.000 

12.12% 

1 1 .86% 

12.28% 

4.93% 

6.02% 

5a  exam 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


14 


70 


61 


5b  exam 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


35 


60 


62 


Quiz 

Scores 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


Unavailable 


68 


76 


74 


46 

61 

50 

70 

65 

109 

34 

44 

54 

37 

30 

50 

34 

28 

44 

10 

16 

49 

17 

16 

83 

9 

12 

54 

14 

17 

77 

12 

19 

65 

25 

29 

119 

30 

38 

115 

16 

56 

160 

48 

69 

122 

78 

44 

110 

8 

14 

117 

633 

715 

1596 

33.316 

37.632 

84.000 

18.25% 

20.61% 

46.01% 
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